PaperlayerView exact revision on arXiv

Source-linked AI summary

FREA: A Multi-Source Expert Benchmark for Reaction Feasibility Verification

Botao Yu, Bo Zhou, Daniel Adu-Ampratwum, Frazier N. Baker, Ziru Chen, Reza Averly, Ye Liu, Wenhao Gao, Xia Ning, Huan Sun

arXiv:2610.06614v1cs.AI

TL;DR

Feasibility verifiers gatekeep reactions proposed for synthesis planning, but their agreement with chemists across candidate sources is unclear. FREA evaluates verifiers against expert labels and finds that strengths vary by source, while generated-negative training gains do not reliably transfer to model proposals.

  • Problem

    It is unclear whether verifiers that appear reliable on recorded or generated reactions agree with chemists on the candidates they must screen.

    Source: §1 · p. 1

  • Method

    FREA benchmarks verifiers on 751 expert-labeled reactions from four sources and pairs evaluation with a generated corpus for matched forward-model training comparisons.

    Source: §1 · p. 2

  • Results

    No verifier leads across sources: criterion-prompted LLMs are competitive, forward models are strongest on retrosynthesis proposals, and generated-negative training gains do not extend to those proposals.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Verifier selection and calibration should use data representative of the intended screening task, and proposal screening is a central test for negative-data design.

    Source: §8 · p. 10

    The mixture-based training comparison leaves open whether the lack of retrosynthesis-proposal improvement is shared across generation methods or depends on the mixture.

    Source: §7.1 · p. 8

Abstract

from arXiv · show

As generative models and AI agents propose chemical reactions at a scale beyond expert review, feasibility verifiers decide which proposals enter synthesis planning. But do their decisions agree with chemists across different kinds of candidates? We introduce FREA, a benchmark of 751 reactions labeled by expert chemists under an explicit feasibility criterion, drawn from retrosynthesis model proposals, zero-yield experimental records, edits by large language models (LLMs), and five negative candidate generation methods. Our evaluation finds that no verifier leads across all sources: LLMs given only the criterion are competitive with dedicated verifiers, while forward models perform best on retrosynthesis proposals but reject most feasible edits of recorded reactions at the evaluated operating points. Looking beyond aggregate scores, both forward models perform below chance when separating infeasible alternative disconnections from feasible generated candidates. To study whether negative supervision addresses these weaknesses, we also release a corpus of over 14 million recorded reactions and generated negative candidates. In matched training comparisons, adding a mixture of generated negatives to forward training raises mean AUROC across sources, but these gains do not extend to retrosynthesis proposals. Varying the generation method further shows that the largest gain on generated candidates coincides with worse proposal screening. These findings motivate evaluating verifiers against experts across sources and designing negatives for transfer to model proposals.

1 INTRODUCTION

FREA tests whether feasibility verifiers agree with chemists across candidate sources, using 751 expert-labeled reactions. Results show source-dependent verifier strengths and weaknesses, while negative-training gains do not consistently transfer to model proposals.

1 INTRODUCTION

  • As automated systems propose reactions beyond expert review, verifiers gatekeep downstream planning; existing evaluation on recorded or generated reactions does not establish agreement with chemists.Published data favor successful outcomes, and generated negative candidates need not be infeasible.Source: §1 · pp. 1–2
  • FREA provides an expert-labeled benchmark of 751 reactions from four sources, including 341 feasible and 410 infeasible cases labeled under an explicit criterion.The benchmark includes recorded failure reasons.Source: §1 · p. 2
  • No verifier leads across sources: criterion-prompted LLMs are competitive, while forward models are strongest on retrosynthesis proposals but reject most feasible recorded-reaction edits.Claude Opus 5 achieved the highest mean balanced accuracy among the evaluated verifiers.Source: §1 · p. 2
  • Both forward models rank infeasible alternative product disconnections below feasible references, with AUROCs of 26.2 and 32.5.Adding generated negatives improves this comparison, but the gains do not extend to retrosynthesis proposals.Source: §1 · p. 2
  • Matched training comparisons using over 14 million recorded reactions and generated negatives find that a mixture raises mean AUROC across sources, but not on retrosynthesis proposals.The largest gain on generated candidates coincides with a decline on model proposals.Source: §1 · p. 2

2 RELATED WORK

Prior work uses feasibility filters in synthesis planning, expert or precedent-based evaluation, and experimental negative-reaction resources. FREA combines multiple candidate sources under a common expert criterion.

2 RELATED WORK

  • Planning systems use classifiers trained on recorded reactions and constructed negatives, but evaluation on generated negatives does not test whether their labels agree with experts.Examples include in-scope filtering, template-generated negatives, and combined scorers reviewed by experts.Source: §2 · p. 2
  • Precedent-based and concurrent expert evaluations address reaction feasibility, while FREA combines model proposals, experimental records, and controlled perturbations under an explicit criterion.This breadth supports diagnosis of source-dependent failures and evaluation of transfer from generated negative training data.Source: §2 · p. 3
  • Published experimental failures are scarce, and existing resources include private records, high-throughput acid–amine couplings, and enzymatic reactions.Experimental failures describe particular conditions; FREA adds zero-yield candidates, model proposals, and constructed reactions for expert annotation.Source: §2 · p. 3

3 REACTION FEASIBILITY TASK DEFINITION

The paper defines feasibility as whether a proposed reaction step is complete and could yield its target products in isolable form under reasonable conditions. Routine auxiliary components may be assumed, but substantive reactants may not be omitted.

3 REACTION FEASIBILITY TASK DEFINITION

  • A verifier predicts whether a reaction is feasible from reactants and products; its input omits conditions, and the product list need not include every reaction product.The label 1 denotes feasible and 0 denotes infeasible.Source: §3 · p. 3
  • Completeness requires listing every substantive reactant needed to produce the target products, while routine auxiliaries such as catalysts, solvents, and bases may be assumed.A Suzuki coupling needs its organoboron partner, whereas hydrolysis water may be left implicit.Source: §3 · p. 3
  • Reactivity requires reasonable compatible conditions that yield isolable target products in one step or one-pot operation; the targets need not be the major products.The supplied passage defines this requirement as the second feasibility condition.Source: §3 · p. 3
  • The rationale is to model a route step executable from its listed starting materials while allowing only routine components and reasonable unstated conditions.Assuming arbitrary missing reactants or extreme conditions would make infeasible labels uninformative.Source: §3 · p. 3

4 THE FREA BENCHMARK

FREA is a 751-reaction expert benchmark spanning practical screening inputs and controlled perturbations. Its source-specific construction and annotation make it possible to test whether verifier performance transfers across candidate types.

4 THE FREA BENCHMARK

  • FREA contains 751 expert-labeled reactions, 341 feasible and 410 infeasible, across four sources that connect practical screening, targeted diagnosis, and transfer evaluation.The sources combine reactions encountered in practice with deliberate perturbations.Source: §4 · p. 4

4.1 FOUR REACTION SOURCES

  • Retrosynthesis proposals from eight single-step models are retained whether or not they reproduce recorded reactants; zero-yield records remain potentially feasible pending expert annotation.The proposal set includes valid alternatives to recorded reactions, while failure under recorded conditions alone does not determine feasibility.Source: §4.1 · p. 4
  • LLMs edit recorded reactions to create superficially coherent intended-infeasible candidates; five generated-reaction methods alter reactants or products while preserving the other side.RA, RR, and NC change reactants; FT and ET change products, with feasibility decided by expert annotation.Source: §4.1 · pp. 4–5, Figure 1 · p. 5

4.2 EXPERT ANNOTATION

  • Two chemists labeled reactions without source or prediction information, using calibration, independent annotation, and joint review of disagreements.Experts judged 79 of 93 zero-yield records feasible and 33 of 289 generated candidates intended as negatives feasible.Source: §4.2 · p. 5

5 EVALUATION PROTOCOL

The evaluation compares criterion-prompted LLMs, forward models, and systems trained on generated negatives, alongside rule-based baselines. It assesses agreement with expert labels using source-level metrics and held-out thresholds.

5 EVALUATION PROTOCOL

  • LLMs receive reactants, products, and the feasibility criterion, while forward models use top-10 product inclusion or target log-likelihood readouts.The evaluated systems also include a rule-based baseline and the AiZynthFinder filter.Source: §5 · pp. 5–6
  • Table 2 reports FREA results by source and their four-source average, with feasible and infeasible counts shown in parentheses.One unparseable Claude response on LP is excluded from that model’s metrics.Source: Table 2 · p. 6
  • Negative-trained systems include a continued BARTSmiles forward model, a direct feasibility classifier, and a descriptor-based gradient-boosted tree.They use the generated corpus training split, which contains no benchmark reaction.Source: §5 · p. 6
  • AUROC is the primary metric, with balanced accuracy covering systems that return decisions without continuous scores.Continuous-score thresholds are fixed on held-out generated pairs; results are reported by source and as an unweighted four-source mean.Source: §5 · p. 6
  • Implementation details and the full evaluation protocol specify inference and training settings, model and threshold selection, and statistical tests.Source: §5 · p. 6

6 RESULTS

No verifier agrees best with experts across all candidate sources: criterion-prompted LLMs are competitive, while forward models excel on retrosynthesis proposals but reject many feasible reaction edits. Within generated candidates, performance varies by construction, and below-chance rankings expose a failure hidden by aggregate scores.

6.1 PERFORMANCE ACROSS SOURCES

  • Claude Opus 5 achieves the highest four-source mean balanced accuracy (68.0), and criterion-prompted LLMs are competitive with dedicated verifiers.Both LLMs reject only 44.7–57.6% of infeasible RP proposals, and each rejects only 4 of 14 infeasible ZR records.Source: §6.1 · p. 6

6.2 PERFORMANCE BY GENERATION METHOD

  • 26.2 AUROC (95% CI [15, 37]) for Chemformer and 32.5 ([20, 45]) for BARTSmiles on NC are below chance against the same 33 feasible GR reactions.Table 3 compares infeasible candidates by generation method against that shared feasible reference; both forward models rank NC infeasible candidates above feasible ones.Source: Table 3 · p. 7, §6.2 · p. 7

6.1 PERFORMANCE ACROSS SOURCES

  • Forward models accept only 28.1–37.5% of feasible LP reactions and 12.1–33.3% of feasible GR reactions at evaluated operating points, versus 94.9–98.7% on ZR.The authors suggest that training to reproduce recorded reactions may penalize feasible departures from recorded patterns; they recommend likelihood scoring over beam search for practical use.Source: §6.1 · p. 7
  • 75.2 mean AUROC for BARTSmiles (LL + neg.) is higher than the direct negative-trained classifier and Descriptor GBDT, but its gains do not extend to RP.The negative-trained model’s gains are concentrated in LP and GR.Source: §6.1 · p. 7
  • No verifier leads across sources; forward models screen retrosynthesis proposals best, but their mean lead with generated negatives does not extend to those proposals.For retrosynthesis proposals, BARTSmiles likelihood scoring has the highest AUROC (78.9), while generated-negative gains come mainly from LP and GR.Source: §6.1 · p. 7

6.2 PERFORMANCE BY GENERATION METHOD

  • NC increases from 32.5 to 94.2 and RA from 46.1 to 72.7 for BARTSmiles (LL + neg.), while RR decreases from 79.8 to 53.4.The higher overall GR AUROC combines gains and losses across methods; the discriminative verifier scores below BARTSmiles (LL + neg.) on four of five methods.Source: §6.2 · pp. 7–8
  • Claude rejects at least 83% of each method’s infeasible candidates, but also rejects two-thirds of feasible GR reactions.The authors note that rejection of infeasible candidates must be considered together with rejection of feasible alternatives.Source: §6.2 · p. 8
  • Source-level scores can conceal failures on particular constructions, and reaction-level performance need not preserve expert rankings of retrosynthesis models.The section leaves open whether negative supervision improves agreement beyond specific constructions, especially on RP.Source: §6.2 · p. 8

7 TRAINING WITH GENERATED NEGATIVES

Matched training comparisons test whether generated-negative supervision transfers across reaction sources, then vary negative-generation methods and formulate three hypotheses for improving transfer. The mean AUROC gain is concentrated outside retrosynthesis proposals, while generation-method rankings and estimated label purity differ in their effects.

7 TRAINING WITH GENERATED NEGATIVES

  • Each training setting uses a single run, so the observed gains are exploratory; the corpus contains no benchmark reaction in training.The study first isolates negative supervision, then varies the generation method.Source: §7 · p. 8

7.1 ADDING NEGATIVE SUPERVISION

  • The matched control shares BARTSmiles initialization, positive reactions, training schedule, and one epoch over 800,000 pairs, isolating the added preference loss.The loss ranks each recorded reaction above its generated candidate while retaining forward-prediction training.Source: §7.1 · p. 8
  • The four-source mean AUROC rises from 71.4 for the continued-training control to 75.2 with negatives, while scores decrease slightly on RP.A 2,400,000-pair repeat preserves the directions, but no within-source differences survive Holm correction; the negative mixture leaves method-specific RP transfer unresolved.Source: §7.1 · p. 8

7.2 CHOOSING THE GENERATION METHOD

  • NC raises GR AUROC by 12.1 points but lowers RP by 2.8; RR raises mean AUROC by 3.2 points, exceeding the mixture’s 2.3.The method comparison varies negative candidates while holding the forward-prediction stream and training schedule fixed.Source: Table 5 · p. 9, §7.2 · p. 9
  • Only FT’s 6.4-point mean-AUROC decline survives Holm correction (padj = 0.001); its estimated negative purity is also lowest, at 77.2%.The decline cannot be attributed to FT’s construction independently of label noise.Source: §7.2 · p. 9

7.3 HYPOTHESES FOR IMPROVING GENERALIZATION

  • Auditing recurring labeling errors and filtering or down-weighting uncertain candidates could test whether preserving feasible alternatives improves transfer.Generated candidates can remain feasible, so assigning them negative labels can penalize valid alternatives to recorded reactions.Source: §7.3 · p. 9
  • Matching recorded reactions and negatives on simple molecular descriptors, while controlling label purity, is proposed as a test of more transferable learning.RR candidates are hardest for the descriptor classifier to distinguish from recorded reactions among RR, NC, and ET, and RR has the largest observed mean gain.Source: §7.3 · p. 9
  • Expert-reviewed development-set proposals could guide perturbations and method weights toward recurring proposal errors, with held-out proposals testing transfer.This hypothesis follows the contrasting NC and mixture outcomes: both improve GR while lowering RP.Source: §7.3 · pp. 9–10

8 CONCLUSION

FREA shows that verifier strengths depend on candidate source, and argues that verifiers should be selected and calibrated on data representative of their intended screening task. Its appendix specifies the feasibility criterion and annotation protocol underlying that expert reference.

8 CONCLUSION

  • No verifier leads across all sources; criterion-prompted LLMs are competitive, while forward models lead on retrosynthesis proposals but fail on some constructed candidates.Matched training experiments show that generated-negative gains vary by generation method and do not extend to model proposals.Source: §8 · p. 10

A RELATED WORK, EXTENDED

  • Prior resources use constructed negatives, experimental failures, precedent, or expert labels in different ways; FREA combines expert-labeled candidates across four sources to evaluate verifiers and training transfer.Precedent-based scoring is complementary because expert judgments can accept sound proposals without a matching recorded precedent.Source: A RELATED WORK, EXTENDED · p. 14

B.1 CRITERION

  • Feasibility requires every substantive reactant and reasonable compatible conditions that yield the target products in isolable form, in one step or a credible one-pot operation.Routine auxiliary species may be assumed, but substantive reactants may not.Source: §B.1 · pp. 15–16

B.2 RATIONALE AND SCOPE

  • The criterion measures expert-judged validity of a retrosynthesis proposal, not whether a transformation is recorded, precedented, or experimentally successful under specified conditions.An infeasible label marks a defect as proposed; it does not claim impossibility under every conceivable condition.Source: §B.2 · p. 17

B.3 DECISION PROCEDURE AND RECORDED FIELDS; B.4 FAILURE REASONS

  • Annotators record feasible, infeasible, or uncertain labels using an ordered decision procedure, with fixed precedence for failure reasons.Reasons distinguish missing substantive reactants from reactivity failures, and competing outcomes can remain feasible if the target is isolable under reasonable conditions.Source: §B.3 · p. 17, §B.4 · p. 18

B.5 NOTES ON FEASIBLE PROPOSALS

  • Notes record assumptions accompanying feasible labels, while the reported evaluation target is the binary feasibility label.Difficulty ratings are not used in reported evaluations.Source: §B.5 · pp. 18–19

B.6 ANNOTATORS AND INTERFACE; B.7 PROTOCOL CALIBRATION

  • Two annotators labeled proposals independently after protocol calibration, with disagreements discussed and the guideline amended before main annotation.The calibration reactions in the pool were settled during calibration and excluded from the agreement analysis.Source: §B.6 · p. 19, §B.7 · p. 19

B.8 AGREEMENT AND ADJUDICATION

  • Independent pre-adjudication agreement was 77.9%, with Cohen’s κ = 0.55 (95% CI [0.49, 0.61]); the benchmark includes 751 reactions after 49 unresolved disagreements were excluded.The benchmark comprises 591 matching independent labels, 119 adjudicated labels, and 41 calibration reactions.Source: §B.8 · p. 20

C CONSTRUCTION OF THE BENCHMARK

The appendix documents how the benchmark and generated corpus were assembled, then specifies the evaluated verifier systems and their training. Reactions were drawn from several sources, screened, and expert-labeled; the system comparisons use distinct forward, rule-based, descriptor, and generated-negative-trained approaches.

D NEGATIVE CANDIDATE GENERATION

  • The corpus starts with curated ORDerly positives and applies five negative-generation methods to each training and test positive, retaining parent links and filtering candidates through 16 quality-control rules.RA, RR, and NC perturb reactants; FT and ET perturb products, and no generated candidate is assumed infeasible without expert annotation.Source: §C.1 · p. 20, §C.2 · p. 21, D NEGATIVE CANDIDATE GENERATION · pp. 21–22

C.3 COLLECTING THE FOUR SOURCES

  • The benchmark combines eight-model retrosynthesis proposals, zero-yield records, LLM edits, and generated candidates; annotators labeled the screened pool, yielding 751 reactions.Proposals were retained whether or not they reproduced recorded reactants; zero yield records only that tried conditions failed, while LLM intent did not determine feasibility labels.Source: §C.3 · p. 21, §C.4 · p. 21

C.5 SPLITS AND OVERLAP

  • All benchmark reactions except zero-yield records derive from the test split, with no benchmark reactant–product pair in model training; product overlap remains possible and the overlap comparison is descriptive.115 benchmark reactions share a main product with training reactions; BARTSmiles scores 76.6 AUROC on 97 such proposals versus 80.8 on the other 185, but the subsets may differ in difficulty.Source: §C.5 · p. 21

D.2 RANDOM ADDITION (RA)

  • The five candidate methods make distinct constructions: RA edits reaction-center reactants, RR substitutes similar or random reactants, and NC proposes alternative product disconnections.FT applies forward templates and ET enumerates elementary-template steps; their unrecorded products are not necessarily infeasible.Source: §D.2 · p. 22, §D.3 · p. 22, §D.4 · p. 23

E SYSTEM IMPLEMENTATIONS

  • Forward models use target recovery or product likelihood, while the released AiZynthFinder filter is evaluated without retraining and the feature classifier uses 58 descriptors.The AiZynthFinder comparison is bounded by different training data and input conventions; potential overlap with its unreleased training set is checked against a superset.Source: §E.1.2 · p. 25, §E.1.3 · p. 25, §E.1.4 · p. 25

E.1.5 RULE-BASED BASELINE

  • The rule-based baseline is an RDKit cascade, but pre-annotation screening already applies its first four rejection rules, leaving only rules five and six able to reject benchmark reactions.It averages 60.2 BAcc over the four sources.Source: §E.1.5 · p. 26

E.2.1 THE DISCRIMINATIVE VERIFIER

  • Discriminative training pairs each generated candidate with its parent recorded reaction and combines pairwise ranking, cross-entropy, and score-centering losses.The ranking term prefers the recorded reaction, cross-entropy anchors scores to a zero decision boundary, and centering controls the batch mean.Source: §E.2.1 · p. 27

E.2 MODELS TRAINED WITH GENERATED NEGATIVE CANDIDATES

  • The seven discriminative verifiers share a Stage-1 checkpoint, Stage-2 loss, optimizer settings, and 20,000-step comparison, while the controlled-pair studies hold forward-training streams and schedules fixed.The discriminative family sees 10.24 million pairs per model; the controlled pairs train for one epoch, with generated candidates either unused or included in a preference loss.Source: §E.2.2 · p. 27, §E.2.1 · p. 27, §E.2.3 · p. 28

E.2.4 THE PER-METHOD COMPARISON, AND ITS PRE-REGISTRATION

The per-method study varies only which generated pairs inform the preference loss while fixing the positive stream and training recipe. Its preregistered predictions were all reflected in the observed point estimates.

E.2.4 THE PER-METHOD COMPARISON, AND ITS PRE-REGISTRATION

  • The seven groups share one forward-prediction stream and differ only in preference-loss pairs: none, 800,000 from one method, or 160,000 from each method.All train for one epoch from the same initialization, with identical optimizer settings and seed.Source: §E.2.4 · p. 28
  • The preregistration fixed the endpoint, paired-bootstrap analysis, purity-based method comparisons, and three predictions; observed point estimates followed all three predictions.The primary endpoint was four-source mean AUROC change from the reference, tested with source-stratified paired bootstrap and Holm correction.Source: §E.2.4 · p. 28
  • The first sample included four benchmark reactions, so runs were stopped and restarted with clean replacement pairs; no final stream shared a reaction with the benchmark.The positive-only reference in this experiment scored 71.7 mean AUROC, versus 71.4 for the smaller control in Table 4.Source: §E.2.4 · p. 28

F.1 METRICS

The evaluation reports within-source metrics because feasibility rates differ sharply across sources and pooling can distort system comparisons. It also checks sensitivity to annotation, checkpoint, threshold, and statistical-testing choices.

F.1 METRICS

  • Pooling can reverse a model comparison: BARTSmiles (disc.) exceeds Chemformer LL in pooled AUROC (82.5 versus 81.9), but trails in four-source mean AUROC (69.7 versus 72.6).A constructed pooled score reaches 82.1 despite tied within-source comparisons, illustrating how source rates can raise pooled AUROC.Source: §F.1 · p. 29

F.2 SENSITIVITY TO AUXILIARY-REAGENT ANNOTATIONS

  • On the subset excluding 104 feasible reactions annotated with missing auxiliary reagent, BARTSmiles (LL + neg.) is comparable to forward baselines, while its advantage over BARTSmiles (disc.) remains 4.2 points.Claude Opus 5 retains the highest mean BAcc, though some intermediate rankings change.Source: §F.2 · p. 29, Table F.1 · p. 29

F.3 CHECKPOINTS

  • BARTSmiles (disc.) remains at or below 67.8 AUROC on RP across checkpoints, and its oracle average of 71.2 remains below forward-model likelihood scores of 72.6 and 72.8.The RR-trained verifier remains best among the seven under fixed-budget, final, and oracle checkpoint rules.Source: §F.3 · p. 30, Table F.3 · p. 31

F.4 THRESHOLDS

  • Continuous-score thresholds are selected to maximize BAcc on balanced held-out generated pairs and then frozen for FREA, with the generation-method mixture potentially affecting the selected threshold.For BARTSmiles LL, freezing the threshold loses 1.8 points on RP, 1.9 on ZR, 7.3 on LP, and 15.0 on GR versus source-tuned thresholds.Source: §F.4 · p. 30, Table F.4 · p. 31

F.5 STATISTICAL TESTING

  • Only one of twelve Holm-corrected paired DeLong comparisons survives correction: BARTSmiles LL exceeds BARTSmiles (disc.) by 11.1 AUROC points on RP.Neither controlled pair reaches significance on any source, including a 9.0-point GR difference with only 33 feasible candidates.Source: §F.5 · p. 30, Table F.2 · p. 30

G ADDITIONAL RESULTS

Across additional analyses, verifier performance varies by candidate construction, and reaction-level scores do not necessarily reproduce expert rankings of retrosynthesis models. LLM-based rankings align most closely in this sample, while thresholded continuous scores show little ranking agreement.

G ADDITIONAL RESULTS

  • Neither evaluated LLM has higher BAcc on LP perturbations from its own model family, and a verifier is often not best on candidates from its training method.For example, the RR-trained verifier scores higher on RA candidates than the RA-trained verifier.Source: G ADDITIONAL RESULTS · p. 32, Table G.4 · p. 33

G.1 RANKING RETROSYNTHESIS MODELS

  • The LLM-based rankings agree most closely with experts: Spearman’s ρ is 0.93 for Claude (95% CI [0.31, 0.96]) and 0.76 for GPT ([0.26, 0.93]).Thresholded continuous-score systems range from −0.19 to +0.17, with intervals spanning zero.Source: §G.1 · pp. 32–33

G ADDITIONAL RESULTS

  • Table G.3 reports each system’s rejected share of infeasible GR candidates and accepted share of the 33 feasible GR candidates at its operating point; both shares depend on threshold placement.Source: Table G.3 · p. 33

G.1 RANKING RETROSYNTHESIS MODELS

  • Strong reaction-level RP scores do not ensure agreement with expert model rankings: BARTSmiles LL has ρ = +0.01 despite RP BAcc of 71.5, near Claude’s 69.9.The ranking analysis covers eight models and sampled proposals, and its wide intervals do not establish a general ranking of verifiers.Source: §G.1 · p. 33
  • BARTSmiles LL retains 50.0% of RP proposals, of which 87.9% are feasible; Claude retains 70.2%, of which 81.8% are feasible.Precision should be considered together with the fraction retained when evaluating a verifier for planning.Source: §G.1 · p. 33

H DATA CARD

FREA releases an expert-labeled benchmark, a generated-negative corpus, and an annotation protocol, with resources intended for verifier evaluation and negative-generation comparisons.

H DATA CARD

  • The benchmark records canonical reactant and product SMILES, feasibility labels, source and generating method, and recorded failure reasons or annotator notes.The generated corpus pairs negative candidates with parent reactions and separates training and test portions with no shared reaction.Source: H DATA CARD · p. 33
  • Table G.5 compares each method’s infeasible candidates with feasible candidates from that same method; cells use only 4–13 feasible examples and are therefore imprecise.Italic cells mark a single-method model evaluated on its own generation method.Source: Table G.5 · p. 34
  • Table G.6 ranks eight retrosynthesis models by proposal acceptance share and reports precision and rank correlation against experts.Acceptance is measured over 282 proposals and compared with the 69.9% expert acceptance rate; intervals use 5,000 bootstrap resamples.Source: Table G.6 · p. 34
  • The release includes the chemists’ annotation guideline, decision procedure, and failure-reason taxonomy.The protocol also includes a documented procedure for annotation decisions.Source: H DATA CARD · p. 34
  • Two author-chemists labeled reactions from curated ORDerly records without seeing their sources; no new experiments were run.The records derive from ORDerly, a curated subset of the Open Reaction Database drawing on patents and published literature.Source: H DATA CARD · p. 34
  • The benchmark supports verifier evaluation, while the generated corpus supports training and comparisons of negative-generation methods.Annotation quality and application scope are documented separately.Source: H DATA CARD · p. 34

I LIMITATIONS

FREA’s labels and comparisons are bounded by expert judgment, selected data and operating settings, with several training findings remaining exploratory.

I LIMITATIONS

  • The feasibility criterion does not test reactions experimentally or specify conditions, so a feasible label does not establish a practical procedure, acceptable yield, or complete-route quality.Route quality can also depend on compatibility between steps and purification.Source: I LIMITATIONS · p. 35
  • Labels reflect two chemists’ judgments: pre-adjudication agreement was 77.9% (κ = 0.55), and joint review cannot rule out errors or represent other experts’ agreement.Binary labels also omit uncertainty in borderline cases.Source: I LIMITATIONS · p. 35
  • All four sources come from ORDerly records and are screened before annotation, while source proportions are evaluation choices rather than deployment estimates.GR shares generation methods with the negative-training corpus, so its gains alone do not demonstrate transfer to model proposals or experimental records.Source: I LIMITATIONS · p. 35
  • Some comparisons rely on small classes, including 14 infeasible ZR reactions and 33 shared feasible references in per-method GR analyses.Each GR per-method AUROC compares against a mixed reference set and may reflect generation-method differences; intervals omit annotation errors and training-run variation.Source: I LIMITATIONS · p. 35
  • Generated candidates may be feasible, and treating them as negatives can penalize valid alternatives; training comparisons use one backbone, corpus, and seed per setting.Most observed gains are not significant after multiple-comparison correction, so hypotheses about label reliability and transfer need broader tests.Source: I LIMITATIONS · p. 35
  • Thresholds calibrated on generated pairs may transfer poorly, and the reported LLM findings cover two models under one prompt.Retrosynthesis rankings cover eight models on sampled proposals at evaluated operating points, not a general ordering of planners or verifiers.Source: I LIMITATIONS · p. 35
Loading 2610.06614v1…