Source-linked AI summary
SureRoute: Toward a Hallucination-Free Self-Improving Platform for Retrosynthesis
Jieli Zhou, Naiwu Chen, Longzhang Liu, Peiyu Zhang
TL;DR
Retrosynthesis models can generate plausible routes without reliably identifying experimentally infeasible ones, and self-improving systems may reinforce such errors. SureRoute addresses this gap with a verifier-anchored platform combining ensemble generation, retrieval, and ChemHarness for executable verification and reliability-first ranking. On 350 industrial targets, it reports higher recall and lower Chemical Hallucination, while ChemHarness generalizes as a model-agnostic reranker.
Problem
Retrosynthesis evaluation commonly emphasizes reference matching, leaving selectivity, competing reactive sites, mechanistic support, and route-level chemical trustworthiness insufficiently assessed.
Method
SureRoute combines multi-model generation, literature retrieval, ChemHarness verification, reliability-first reranking, and an evolution layer that learns from confirmed failures.
Results
SureRoute achieves 2.2–3.5× the top-1 recall of SOTA single-step models and frontier LLMs, with 4.6% Chemical Hallucination, while ChemHarness drives detectable top-1 hallucination toward near-zero across arbitrary backbones.
Takeaways & Limitations
Executable, auditable verification can provide a model-agnostic reliability layer for self-improving retrosynthesis without relying exclusively on wet-lab validation.
Takeaways & Limitations
Reranking leaves a 6.9% residual hallucination rate for Chemformer when its entire top-10 candidate pool contains only competition-flagged disconnections.
Abstract
from arXiv · showhide
AI models, including large language models, are increasingly integrated into scientific discovery workflows, yet they remain prone to hallucination. In experimental sciences, such errors translate directly into failed wet-lab validations and wasted resources; in self-improving agentic systems, confident errors risk being reinforced rather than corrected. Retrosynthesis provides a representative example of this failure mode: existing models can generate chemically plausible routes, but cannot reliably determine which routes are experimentally feasible. We define \textbf{Chemical Hallucination} as a route that appears valid yet fails under competing reactive sites, unresolved selectivity, or missing mechanistic support, a failure largely invisible to the Recall@$K$ metric. We introduce \textbf{SureRoute}, a chemical verifier-anchored retrosynthesis platform that suppresses Chemical Hallucination. SureRoute combines a multi-model ensemble, data asset retrieval, and \textbf{ChemHarness}, an executable chemical intuition engine for route verification and reliability-first ranking. On a benchmark of 350 real-world industrial targets, SureRoute reaches 74.3\% recall@1, 2.2--3.5$\times$ that of seven single-step models and three frontier LLMs, while cutting top-1 Chemical Hallucination to 4.6\%, a 4--6$\times$ reduction relative to frontier LLMs. As a model-agnostic reranker, ChemHarness drives detectable hallucination toward near-zero across arbitrary backbone candidates. SureRoute shows that reliable scientific AI requires not only strong generation, but executable verification.
1 Introduction
SureRoute frames retrosynthesis as a chemical-verification problem, addressing hallucinated routes that recall@K does not detect. Its verifier-anchored platform combines generation, retrieval, ChemHarness, and reliability-focused reranking.
- The verification problem: Self-improving systems can reinforce confident chemical errors when models generate training signals from their own search results without an external verifier.Wet-lab validation is trustworthy but slow and expensive, motivating executable computational checks.
- The verification problem: Chemical Hallucination describes superficially valid retrosynthetic disconnections that fail under competing reactive sites, unresolved selectivity, or weak mechanistic support.Such failures can remain invisible to recall@K because a model may match reference reactants while ranking chemically dangerous alternatives highly.
- SureRoute: ChemHarness checks candidate routes for functional-group competition, regio-/chemoselectivity risk, mechanistic plausibility, and literature support.SureRoute uses verification to filter hallucinations and ranking to prioritize verified routes.
- SureRoute: SureRoute combines a multi-model intelligent ensemble, literature-precedent retrieval, ChemHarness, and an evolution layer that learns from expert-confirmed failures.The platform is positioned as a verifier-anchored self-improving system rather than another single-step predictor.
- Evaluation: ChemHarness is model-agnostic: applied post-hoc to arbitrary backbones’ top-10 candidates, it drives detectable top-1 Chemical Hallucination toward near-zero.The evaluation covers seven single-step models and three frontier LLMs on 350 real-world industrial molecules.
2 Related Work
Related work spans retrosynthesis prediction, search, self-improvement, and scientific-AI evolution. Across these threads, the unresolved issue is route-level executable verification rather than merely generating or matching candidate reactions.
- Retrosynthesis prediction: Single-step retrosynthesis methods include template-based, template-free, graph-based, and retrieval/alignment-style approaches, which SureRoute pools behind a unified candidate-proposal interface.SureRoute leaves organization and filtering to its search and verification layers.
- Planning and search: Multi-step retrosynthesis research uses search methods such as Monte Carlo Tree Search and AND-OR-tree best-first search to guide route construction.These methods improve expansion reuse, pruning, or search quality through learned guidance.
- Self-improving retrosynthesis: Self-improving retrosynthesis feeds search outcomes back into training, following Expert Iteration and AlphaGo Zero-style policy improvement.Examples include Self-Improved Retrosynthetic Planning, PDVN, and ReSynZ.
- Self-improving retrosynthesis: Recent methods increasingly use verifiable rewards, but verification remains largely single-step rather than route-level, executable, and competition-aware.SureRoute and ChemHarness target this gap by making the self-improvement signal closer to wet-lab semantics.
- Scientific-AI evolution: Scientific-AI self-evolution spans weight, memory, tool/code, and physical closed-loop paths, each with different verifier costs and trust levels.ChemHarness is presented as a chemistry-domain analogue of executable tool/code evolution, while physical experiments offer stronger verification at higher iteration cost.
- Scientific-AI evolution: Across these paths, the bottleneck is shifting from generation capability to verification capability because untrusted self-evolution can amplify bias and hallucination.SureRoute addresses this with auditable computational verification and accumulated failure cases rather than physical validation for every iteration.
3 Method
SureRoute is a four-layer retrosynthesis platform that combines diverse candidate generation, executable chemical verification, reliability-first ranking, and self-improvement from confirmed failures. Its model-agnostic ChemHarness layer prioritizes chemically supported routes over raw model confidence and enables auditable computational feedback.
- Platform architecture: SureRoute combines generation, verification, ranking, and evolution layers, with each addressing a failure mode that the preceding layer cannot solve alone.Generation builds candidate diversity; verification checks chemical reliability; ranking selects verified routes; evolution incorporates expert-confirmed failures.
- Candidate generation: Multiple retrosynthesis models contribute top-10 candidates that are pooled and deduplicated into a broader RRF-ranked candidate set.Candidates are canonicalized before deduplication, and rank information is aggregated across architectures.
- Chemical verification: ChemHarness evaluates competition, selectivity, mechanistic plausibility, and literature precedent, producing a reliability profile for each candidate route.Routes with unresolved reactive-site competition or lacking mechanistic support are treated as chemically unreliable even when syntactically valid.
- Reliability-first ranking: Literature hits and competition-free routes outrank flagged routes, while precedent score and ensemble confidence serve as later ranking criteria.Raw ensemble confidence is retained only as the residual tie-breaker after chemical and precedent-based criteria.
- Pluggability and self-improvement: ChemHarness reranking applies to candidate pools from any backbone without retraining and drives top-1 Chemical Hallucination toward zero in tested model and LLM pools.Its checks operate on the proposed reaction rather than on the identity of the generating model.
- Pluggability and self-improvement: Chemist overrides and downstream disagreements are logged as confirmed failures, which are triaged into rule families to improve coverage and preserve auditability.Explicit, human-readable rules allow chemists to inspect, accept, or correct the rule that fired.
4 Benchmark: industrial retrosynthesis evaluation set
The benchmark evaluates retrosynthesis on 350 realistic industrial molecules using expert-validated references and a unified protocol. It measures both reference-matching recall and the chemical trustworthiness of top-ranked routes under standardized comparisons.
- Dataset: 350 real industrial molecules form the evaluation set, covering realistic scaffolds, multiple functional groups, and substrate-specific constraints.The targets are intended to require chemical judgment rather than pattern matching alone.
- Evaluation protocol: Each target is paired with expert-validated reference routes for comparison with model-generated candidates.The protocol evaluates candidate routes against references collected for every target.
- Evaluation protocol: Each system proposes up to 5 ranked single-step retrosynthetic reactant sets per target.The candidate limit is shared across systems under the unified evaluation protocol.
- Matching: A prediction counts as a hit only when its reactive-core reactant set matches the standardized expert reference.Reagents, solvents, catalysts, salts, and other non-core components are handled consistently across methods.
- Chemical reliability: The evaluation supplements recall@K with expert annotation of chemical reliability risks in the top-ranked route.This distinguishes reference-like predictions from recommendations considered chemically trustworthy.
- Fair comparison: All models and SureRoute variants use the same targets, candidate limit, standardization pipeline, and scoring criteria, with literature overlaps excluded when needed.This controls formatting and direct-database-overlap effects in the comparison.
5 Experiments
The experiments evaluate recall@K and top-1 Chemical Hallucination across specialized retrosynthesis models, frontier LLMs, and SureRoute, including ChemHarness reranking on arbitrary backbones.
- Chemical Hallucination rate: 18.9–27.4% top-1 Chemical Hallucination for frontier LLMs is higher than the 14–20% rate for single-step models.The reported comparison separates raw recall from chemical selectivity awareness.
- Chemical Hallucination rate: 4.6% top-1 Chemical Hallucination for SureRoute is below the 14–20% rate of all seven single-step models.The metric counts routes with functional-group competition or mechanistic implausibility.
- Pluggability: ChemHarness reranking uses each backbone’s own top-10 candidates without changing the candidate pool or underlying model.The reranking signal is based on competition and plausibility.
- Pluggability: ChemHarness reranking drives detectable top-1 Chemical Hallucination toward near-zero across specialized models and frontier LLMs.This includes LLMs whose models were not used to develop ChemHarness’s rules.
- Pluggability: A 6.9% residual rate remains for Chemformer because its top-10 pool can contain only competition-flagged disconnections.Residual hallucination therefore diagnoses whether the candidate pool contains a chemically sound alternative.
5.4 Two selectivity case studies
Two case studies show specialized models and frontier LLMs converging on selectivity-unsafe disconnections, while ChemHarness identifies the chemical risks and favors literature-supported routes.
- Case studies: SureRoute ranks literature-supported routes above selectivity-unsafe alternatives in both case studies.The figure descriptions identify the ester-hydrolysis and pyrimidinone routes as the retained or top-ranked literature routes.
- Case 1: chemoselectivity: In the Williamson case, the carboxylate competes with the phenoxide under the basic alkylation conditions required for ether formation.The literature route avoids this ambiguity by installing the ether earlier and unmasking the acid through ester hydrolysis.
- Case 1: chemoselectivity: Every specialized model and two of three frontier LLMs propose the chemoselectively unsafe Williamson disconnection as top-1.ChemHarness flags these routes for O-alkylation competition.
- Case 2: regioselectivity: In the dichloropyrimidine case, coupling at the more activated C–Cl site is expected to produce the undesired regioisomer.ChemHarness flags the proposed disconnection for regioselectivity risk, independently annotated by an XtalPi chemist.
- Case 2: regioselectivity: All seven specialized models and Claude Opus 4.8 rank the dichloropyrimidine disconnection first, while GPT-5.5 gives an unmatched answer and DeepSeek V4 returns no route.No backbone recovers the literature disconnection in this case.
Further Chemical Hallucination cases
Additional cases show recurring selectivity hallucinations across models, while ChemHarness-based reranking anchors the top-1 prediction on literature routes and supports multi-step recovery.
- Additional cases: All seven single-step models and, in most cases, one or more LLMs converge on a single flagged disconnection in the additional cases.The recurring pattern spans independently trained or prompted systems.
- Additional cases: Competing bromides plus an unprotected amine create a selectivity problem in the dibromo-aminopyrazine Suzuki coupling.The literature route instead disconnects at one unambiguous aromatic-ring position.
- Additional cases: Competing S- versus N-methylation makes the thiourea product structure ambiguous.The literature route uses cyclization to fix the reactive site.
- Additional cases: ChemHarness-based reranking places the literature route in the top-1 slot for each additional case.The cases are summarized in Figure 8 as selectivity-hallucination examples.
- Multi-step recovery: Step-wise ChemHarness reevaluation recovered complete 3–5-step literature routes, with the correct disconnection ranked first at nearly every step.This multi-step verification used a separate set of literature targets with documented experimental routes.
6 Conclusion
SureRoute combines multi-model generation and literature retrieval with ChemHarness, a mechanism-aware verifier that screens retrosynthetic candidates for chemical reliability. On a 350-molecule benchmark, it improves recall while substantially reducing Chemical Hallucination, and ChemHarness generalizes as a model-agnostic reranker.
- Platform: SureRoute pairs a multi-model disconnection ensemble and literature retrieval with ChemHarness, a rule-based, mechanism-aware reliability layer.ChemHarness explicitly screens for functional-group competition and mechanistic implausibility.
- Results: 2.2–3.5× higher top-1 recall than SOTA single-step models and frontier LLMs was achieved on an internal 350-molecule benchmark.The comparison covers both specialized retrosynthesis models and frontier language models.
- Results: 4.6% Chemical Hallucination, a 4–6× reduction relative to frontier LLMs, was achieved on the same benchmark.This result emphasizes reliability beyond reference-matching performance.
- Generalization: ChemHarness reranking drives detectable top-1 Chemical Hallucination toward near-zero across arbitrary backbone candidates, including untuned LLMs.The result supports ChemHarness as a model-agnostic, pluggable reliability layer.
- Implication: SureRoute frames verification capability as central to trustworthy self-improvement because self-graded outputs can compound rather than correct mistakes.Its verifier is intended to support rule refinement from real production failures without physical validation or self-graded reward.
A LLM Evaluation Protocol
The evaluation uses a unified prompt template for frontier LLM retrosynthesis and processes their outputs through the same standardization and matching pipeline used for specialized models.
- Evaluation setup: Frontier LLMs are evaluated with a unified prompt template and the same downstream standardization and matching pipeline as specialized retrosynthesis models.The full prompt contains a system instruction and a user message with two in-context demonstrations followed by the query.
A.1 Prompt Template
The prompt asks an LLM to generate up to five ranked, single-step retrosynthetic reactant sets in SMILES format, using amide-coupling and Suzuki-coupling demonstrations.
- Instruction: The system instruction frames the model as an expert organic chemist proposing reactant sets for one-step synthesis of a target molecule.The task is single-step retrosynthesis rather than complete multistep route planning.
- Demonstrations: Two demonstrations illustrate amide coupling and Suzuki coupling by pairing target structures with alternative reactant sets.The examples use reactant SMILES for carboxylic-acid or acid-chloride coupling and boron-based or boronate Suzuki partners.
- Query: The query requests up to 5 distinct single-step retrosynthetic disconnections for the target.This produces a ranked candidate list rather than a single answer.
- Output format: Each output line must contain one dot-separated reactant set, ordered from most to least likely, with only purchasable-style reactant SMILES and no explanations.The formatting rules exclude reaction arrows and prose.
- Template: The target is supplied as a SMILES placeholder followed by an answer field for generated reactant candidates.The template represents the target using {target SMILES}.
- Evaluation role: Ranked candidate lists enable direct comparison between LLMs and specialized retrosynthesis models under recall@K evaluation.The shared format supports evaluation across multiple candidate positions.
B Target Visualization
Figure 10 visualizes the targets in the industrial retrosynthesis evaluation set.
- Figure: Figure 10 provides a target visualization for the industrial retrosynthesis evaluation set.The passage identifies the figure’s subject but does not specify additional axes, encodings, or comparisons.