PaperlayerView exact revision on arXiv

Source-linked AI summary

RxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature

Jiaxing Sun, Junyuan Gao, Ruikang Zhang, Hao Zheng, Lingli Ge, Xiao An, Jiang Wu, Huazheng Zeng, Jiahe Song, Chen Zhu, Haote Yang, Jingchao Wang, Lu Chen, Yuanyuan Cao, Ruijie Zhang, Shengxin Lu, Weijia Li, Lijun Wu, Conghui He

arXiv:2610.32221v1cs.AI

TL;DR

Automated extraction of structured reactions from Screen Tables and Substrate Scopes remains challenging, and existing benchmarks do not systematically assess this task. RxnExBench provides a reaction-level benchmark with a unified schema and reasoning taxonomy; evaluations show persistent integration and reasoning bottlenecks, even with perfect molecular recognition.

  • Problem

    Screen Tables and Substrate Scopes contain reaction information in dense figures, but VLM performance on structuring reactions from them has not been systematically evaluated and existing benchmarks have coverage or evaluation limitations.

    Source: §1 · pp. 1, 3

  • Method

    RxnExBench benchmarks reaction-level extraction with a 26-field schema, three reasoning-driven difficulty levels, and Oracle-MolRec to separate molecular recognition from higher-level reasoning.

    Source: §1 · p. 3, §4.5 · p. 11

  • Results

    Across 9 VLMs, multimodal integration and chemistry-aware reasoning remain bottlenecks; with Oracle-MolRec, Criterion 1 best F1 rises from 23.16% to 42.67%, while stricter criteria remain limited.

    Source: §1 · p. 4, §4.5 · p. 11, §5 · p. 12

  • Takeaways & Limitations

    The reaction-level schema is designed to support downstream AI-for-Chemistry tasks, including product prediction, yield estimation, condition recommendation, and retrosynthetic planning.

    Source: §3.1 · p. 5

    The dataset focuses primarily on English-language organic synthesis literature and uses relatively strict reaction-level matching, so results are a snapshot rather than an upper bound on task performance.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Screen Tables and Substrate Scopes are information-dense figures widely used in synthetic chemistry literature, yet automated extraction of structured reaction data from them remains challenging. We introduce RxnExBench, a multimodal benchmark for fine-grained reaction-level evaluation. Built from high-impact chemistry journals, it contains 3,565 expert-annotated reactions and 2,361 molecular structures. The dataset adopts a unified schema with 15 reaction roles and 26 core fields, supporting downstream AI-for-Chemistry tasks such as retrosynthetic planning, yield prediction, and reaction condition recommendation. RxnExBench further provides a reasoning-driven difficulty taxonomy, a modular annotation pipeline, and the Oracle-MolRec protocol, which disentangles molecular recognition from higher-level reasoning. Evaluation of 9 state-of-the-art VLMs shows that performance degrades substantially as schema complexity and reasoning demands increase. Under the Oracle-MolRec setting, results further reveal that accurate molecular recognition alone is insufficient, with multimodal integration and chemistry-aware reasoning remaining the primary bottlenecks. We release all data and code to facilitate future research in chemistry-aware multimodal understanding and AI-for-Chemistry dataset construction. The dataset and code will be released at https://github.com/opendatalab/RxnExBench.

1 Introduction

Screen Tables and Substrate Scopes hold valuable reaction information but remain underexplored and difficult to extract because they combine dense visual content with chemistry-aware reasoning demands. RxnExBench addresses this gap with a reaction-level schema and difficulty taxonomy, and its VLM evaluation identifies multimodal integration and higher-level reasoning as persistent bottlenecks.

1 Introduction

  • The benchmark targets a data gap: reaction information is abundant in scientific literature but often appears in unstructured figures and tables unsuitable for direct model training.Existing AI-for-Chemistry work has advanced reaction prediction, condition optimization, and retrosynthetic planning, while high-quality reaction datasets remain scarce.Source: §1 · p. 1
  • Screen Tables and Substrate Scopes remain underexplored despite their prevalence, and extracting their dense figures requires visual understanding, OCR, chemical interpretation, and cross-modal reasoning.Screen Tables compare related reactions under varying conditions; Substrate Scopes show outcomes across reactants under fixed conditions.Source: §1 · pp. 2–3
  • RxnExBench evaluates reaction-level extraction with a unified 26-field schema and three chemistry-grounded difficulty tiers, from local reading to R-group-aware spatial reasoning.The tiers empirically correlate with VLM error rates.Source: §1 · p. 3
  • A modular annotation pipeline, executed by 40 chemistry-trained annotators with three rounds of cross-validation, yields 3,565 reactions, 2,361 molecular structures, and 53,659 annotated fields.The workflow includes template annotation, instantiation, and molecule drawing and canonicalization.Source: §1 · pp. 3–4
  • VLMs struggle with holistic chemistry-grounded integration as schema demands grow, and higher-level reasoning remains the dominant bottleneck even with perfect molecular recognition.The benchmark evaluates 9 state-of-the-art VLMs using Oracle-MolRec to separate molecular recognition from higher-level reasoning.Source: §1 · p. 4

2 Related Work

Prior reaction-extraction research has focused mainly on text, molecular recognition, and reaction schemes, leaving Screen Tables and Substrate Scopes comparatively underexplored. Existing benchmarks cover only parts of these formats or assess structure comprehension rather than complete reaction extraction.

2 Related Work

  • Automated extraction research has predominantly addressed text-based reaction information, while image-based work has targeted general reaction diagrams rather than ubiquitous Screen Tables and Substrate Scopes.Text-based efforts include deep learning and LLM-driven techniques.Source: §2 · p. 4
  • Existing benchmarks leave gaps in full reaction reconstruction: MERMES focuses on conditions and one substrate, while ChemEAGLE covers limited reaction roles and ChemTable emphasizes table structure.These limitations constrain evaluation of complete reaction records and holistic multimodal integration.Source: §2 · p. 4
  • MerMaid evaluates R-group-dependent reasoning in Screen Tables but does not cover Substrate Scopes or multi-condition comparison.It includes SMILES fields for molecular-to-SMILES conversion.Source: §2 · p. 4

3 RxnExBench

RxnExBench evaluates VLM extraction of structured reactions from Screen Tables and Substrate Scopes. It combines a reaction schema, difficulty-aware examples, expert annotation, and reaction-level evaluation.

3 RxnExBench

  • RxnExBench targets VLM understanding and extraction of structured reaction information from Screen Tables and Substrate Scopes.The benchmark focuses on these two chart types in synthetic chemistry literature.Source: §3 · p. 4

3.1 Structured Reaction Schema

  • The reaction schema organizes each reaction into reactants, products, and stages, with stage-associated conditions and molecular-participant details.It includes fields such as SMILES, reactant amounts, and product yields, and is intended to support downstream AI-for-Chemistry tasks.Source: §3.1 · p. 5

3.2 Difficulty-Aware Image Collection and Curation

  • The difficulty levels progress from sequential row or substrate processing, through cross-region information integration, to chemistry-aware R-group inference.Level 2 includes searching distant regions such as footnotes; Level 3 reconstructs full molecular identities from variable R-groups.Source: §3.2 · p. 6
  • The benchmark was curated from approximately 5,000 papers in eight chemistry journals and narrowed to 90 Substrate Scope and 176 Screen Table figures.The selected figures cover all three difficulty levels.Source: §3.2 · pp. 6–7

3.3 Data Annotation Pipeline

  • The annotation pipeline separates reaction-template annotation, template instantiation, and molecular-structure annotation.Forty chemistry-trained annotators completed three cross-validation rounds, producing 3,565 reactions and 2,361 canonical SMILES strings.Source: §3.3 · p. 7

3.4 Evaluation Task Design

  • VLMs must return all image reactions as one schema-compliant JSON object, evaluated with reaction-level precision, recall, and F1 under three exact-match criteria.The criteria progressively include reactants, products, and reagents; add catalyst and solvent; then require exact matches on a broader set of reaction conditions.Source: §3.4 · p. 8

4 Experiments

The experiments show that structured reaction extraction remains difficult for state-of-the-art VLMs, with performance weakening as schema and reasoning demands increase. Reaction-count and error analyses identify molecular recognition, condition extraction, and layout-aware reasoning as persistent challenges, while Oracle-MolRec shows that perfect molecular recognition is not sufficient.

4.1 Model Selection and Experimental Setup

  • The evaluation covers nine state-of-the-art VLMs, including open-source models and commercial APIs.The evaluated models include Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.5, GPT-4o, Doubao Seed 2.0 Pro, GLM-5V Turbo, Intern-S1, Llama 4 Maverick, and Qwen3-VL-235B.Source: §4.1 · p. 8

4.2 Overall Performance and Preliminary Findings

  • Under Criterion 1, GPT-5.5 leads at 23.16% F1, followed by Doubao Seed 2.0 Pro at 19.81% and Gemini 3.1 Pro at 19.47%; the strongest model reaches 8.86% on Criterion 3.The remaining six models cluster below 6% on Criterion 1.Source: §4.2 · p. 8
  • On Criterion 1, Gemini 3.1 Pro scores 40.42% F1 on Level 1 data and 15.24% on Level 3, illustrating the stepwise difficulty-related decline.Figure 5 summarizes average performance across all nine models and three difficulty levels.Source: §4.2 · p. 8, Figure 5 · p. 9
  • Across models, F1 declines from Criterion 1 to Criterion 3 as evaluation requires more schema fields.Figure 5 presents average F1 across difficulty levels and evaluation criteria.Source: §4.2 · p. 8, Figure 5 · p. 9

4.4 Error Analysis

  • Incorrect molecular structures account for 75.51% of extraction errors, while condition extraction and layout reasoning are also identified as coupled bottlenecks.Models performing well on Criterion 1 but degrading on Criterion 3 often miss or merge condition fields; all models undercount reactions on CLS figures.Source: §4.4 · pp. 9–11

4.3 Reaction Count as a Proxy for Basic Understanding Ability

  • VLMs generally match reaction counts in SLS figures, but even the best model reaches only 60% consistency on CLS figures, where models often undercount.Figure 6 compares predicted and ground-truth reaction counts for SLS and CLS figures.Source: §4.3 · p. 9, Figure 6 · p. 10

4.4 Error Analysis

  • Error profiles differ by model: 84.7% of Claude Opus 4.6 erroneous reactions involve SMILES errors, while Field Missing is Llama 4 Maverick’s largest independent component at 29.5%.Intern-S1 shows substantial overlap among all three error categories.Source: §4.4 · p. 9, Figure 7 · p. 10

4.5 Disentangling Molecular Recognition from Higher-Level Reasoning

  • With Oracle-MolRec-IDT, best F1 rises from 23.16% to 42.67% on Criterion 1, yet Criterion 3 reaches only 15.88% and 4/9 models stay below 10%.The authors conclude that extraction also requires holistic multimodal perception, long-range integration, and chemistry-aware reasoning.Source: Table 4 · p. 11, §4.5 · p. 11

5 Conclusion

RxnExBench brings together reaction-level coverage of Screen Tables and Substrate Scopes with a reasoning-driven difficulty taxonomy. Evaluation of nine VLMs shows continued difficulty, particularly with multimodal integration and chemistry-aware reasoning.

5 Conclusion

  • RxnExBench jointly covers Screen Tables and Substrate Scopes with full reaction-level schema, R-group resolution, CLS figures, and reasoning-driven difficulty levels.The benchmark contains 3,565 expert-annotated reactions and 2,361 molecular structures.Source: §5 · p. 11
  • The benchmark uses a 26-field reaction schema across three reasoning-driven difficulty levels.Source: §5 · p. 12
  • Experiments on nine state-of-the-art VLMs show continued difficulty, particularly in multimodal integration and chemistry-aware reasoning; perfect molecular recognition alone is insufficient.Source: §5 · p. 12

Limitations

RxnExBench addresses structured reaction extraction from dense chemical figures, while the benchmark’s current scope is limited to English-language organic synthesis literature and strict reaction-level matching. The appendix describes how Screen Tables and Substrate Scopes encode reaction information and defines the structured schema.

Limitations

  • The dataset primarily covers English-language organic synthesis literature and uses relatively strict reaction-level matching criteria.The authors characterize reported results as a snapshot of general-purpose VLM capabilities, not the task’s upper performance bound.Source: Limitations · p. 12

A.1 Introduction of Screen Tables and Substrate Scopes

  • Screen Tables compare similar reactions under controlled variables, whereas Substrate Scopes show outcomes for varied reactants under fixed conditions.Screen Table entries may require combining row-specific conditions with a shared template; Substrate Scope structures instantiate variable substituents in the template product.Source: §A.1 · pp. 16–17

A.2 Structured Chemical Reaction Schema

  • The Structured Reaction Schema organizes reaction roles and fields as dictionaries, using string or list data types as appropriate.Lists accommodate fields or roles with multiple instances.Source: §A.2 · p. 17

B Data Collection

The data collection section identifies the chemistry journals used and points to tables describing the reaction schema and per-image annotation statistics.

B Data Collection

  • RxnExBench data come from 8 high-impact chemistry journals.The journal list is presented in Table 6, but its entries are not included in the supplied passage.Source: B Data Collection · p. 17
  • Table 5 presents the structured chemical reaction schema of RxnExBench.The supplied caption identifies the table’s subject but does not provide its schema contents.Source: Table 5 · p. 18
  • Table 7 presents average per-image annotation statistics.The supplied caption does not include the statistics themselves.Source: Table 7 · p. 18

C.1 Statistics Related to Data Annotation

RxnExBench comprises 266 images and 3,565 reactions, with 2,361 unique canonical SMILES after deduplication. Its annotation workflow separates template interpretation, instance construction, and molecular structure annotation, supported by web-based platforms.

C.1 Statistics Related to Data Annotation

  • 3,565 reactions across 266 images yielded 53,659 annotated fields and 10,296 molecule occurrences, corresponding to 2,361 unique canonical SMILES after deduplication.The structures contain 130,290 atoms and 134,840 chemical bonds; molecule occurrences may repeat across reactions.Source: §C.1 · pp. 18–19

C.2 Detailed Data Annotation Process

  • The workflow separates reaction-template annotation, template instantiation, and molecular SMILES annotation, with localization and instance-reference preparation supporting those stages.Annotators register reaction instances, fill in context-dependent fields, and associate molecular structures with instances before RDKit standardizes SMILES and inserts them into the final ground truth.Source: §C.2 · pp. 19–20

C.3 Data Annotation Platform

  • Purpose-built web platforms coordinate parallel annotation and support quality checks and data processing.The appendix identifies interfaces for localization, template annotation, reference pre-annotation, instantiation, and molecular SMILES annotation.Source: §C.3 · p. 20

C.4 Data Annotation Team

The annotation team comprised chemistry-trained annotators working under cross-checking and review procedures. Human experts scored below perfect accuracy, and their performance declined with increasing difficulty and stricter criteria; the best evaluated VLM remained far behind.

C.4 Data Annotation Team

  • 40 professional chemical annotators with chemistry-related backgrounds completed knowledge assessments and protocol training, with samples cross-checked and disagreements resolved through discussion and expert review.The modular pipeline separates template annotation, instantiation, and molecular structure annotation to help reduce ambiguity and improve consistency.Source: §C.4 · p. 20
  • Annotators were compensated at or above prevailing skilled-work wage standards, and all provided informed consent after receiving written study and release information.The information included the planned CC-BY-4.0 release and the right to withdraw.Source: §C.4 · p. 20

C.5 Human Performance Upper Bound

  • 91.33%, 85.27%, and 81.66% were the experts’ overall F1 scores under Criteria 1, 2, and 3, respectively.The authors treat these scores as an empirical performance ceiling for the benchmark, noting interpretive ambiguity in footnotes, abbreviations, stereochemistry, and tautomer choice.Source: §C.5 · p. 21, Table 8 · p. 21
  • Under Criterion 1, expert F1 declined from 98.98% on Level 1 to 93.49% on Level 2 and 87.68% on Level 3; on Level 1, it fell from 98.98% under Criterion 1 to 94.92% under Criterion 3.The authors say these patterns support the reasoning-driven taxonomy and indicate added difficulty as required-match fields increase from three to ten.Source: §C.5 · p. 21
  • 23.16% F1 was the best VLM’s score under Criterion 1, 68.17 points below human performance; under Criterion 3, the gap was 72.80 points (8.86% vs. 81.66%).The authors conclude that fully automated structured reaction extraction from these figures remains an open problem.Source: §C.5 · p. 21

D Examples of RxnExBench Annotation Result

The section shows example Screen Table and Substrate Scope annotations and documents the prompts used for standard and Oracle-MolRec inference.

D Examples of RxnExBench Annotation Result

  • Figures 15–16 show example Screen Tables and Substrate Scopes alongside their detailed reaction annotations in Figures 27–30.The section identifies the paired figures as examples and corresponding annotation results.Source: D Examples of RxnExBench Annotation… · pp. 21–22

E Inference Prompt

  • The inference prompts cover standard settings and three Oracle-MolRec variants for both Screen Tables and Substrate Scopes.The variants are Oracle-MolRec-IDT, Oracle-MolRec-BBOX-IDX, and Oracle-MolRec-BBOX-CORD.Source: E Inference Prompt · p. 22

D Examples of RxnExBench Annotation Result

  • Table 10 reports VLM recall scores on RxnExBench for the full dataset and three difficulty levels under three evaluation criteria; Table 9 provides the criterion details.Boldfaced numbers denote best performance, while underlined numbers denote runner-up.Source: Table 10 · p. 22, Table 9 · p. 22

F Criterion and evaluation details of RxnExBench

RxnExBench evaluates exact reaction-field matches under criteria of increasing field coverage and normalizes molecular and text representations before comparison.

F Criterion and evaluation details of RxnExBench

  • A true positive requires an exact match to ground truth across every field required by the selected criterion.Criterion 2 adds catalyst and solvent fields to Criterion 1; Criterion 3 checks ten fields.Source: F Criterion and evaluation details of… · pp. 22–23
  • Molecular structures are compared after RDKit canonicalization, with equivalence requiring identical canonical molecular graphs and fingerprint Tanimoto similarity of 1.0.The similarity is computed on RDKit fingerprints; the passage defines A and B as binary fingerprints.Source: F Criterion and evaluation details of… · pp. 23–24
  • Other string fields are lowercased and normalized for spacing and common symbols or delimiters before matching.The procedure is intended to tolerate minor formatting differences without relaxing field semantics.Source: F Criterion and evaluation details of… · p. 23
  • Tables 13 and 14 are identified as reporting RxnExBench results for MerMaid and ChemEAGLE, respectively.The supplied captions do not provide result values.Source: Table 13 · p. 24, Table 14 · p. 24

G Detailed experimental results

The experimental results are organized by inference setting, evaluation criterion, and difficulty, with Oracle-MolRec variants outperforming standard inference on the MolIDT subset.

G Detailed experimental results

Loading 2610.32221v1…