ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning

1 Introduction

• Existing chemistry benchmarks test parts of the reasoning process, but not the complete chain from perceiving a structure to attributing an effect to a feature.The paper calls the final decision explanatory attribution and notes that a model can predict the correct product while relying on the wrong rationale.Source: §1 · p. 1
• A multi-agent framework retrieves and validates structure-to-effect evidence, applies quality control, and preserves each selected question’s source-paper provenance.The framework constructs ChemExplain from about 5,000 chemistry papers; the correct relation comes from supporting source evidence rather than being invented by the question-generation LLM.Source: §1 · pp. 1–2
• ChemExplain contains 1,180 multiple-choice questions from 383 peer-reviewed papers, with correct feature–mechanism relations grounded in source-paper evidence.Questions pair a molecular or reaction image and observed effect with four explanations linked to different visible structural features.Source: §1 · p. 2
• 71.4% of the strongest model’s errors are attribution failures, and attribution is the largest error category for the open-source and chemistry-specific models analyzed.The evaluation covers 12 proprietary, open-source, and chemistry-specific VLMs; the strongest reaches 78.6% accuracy.Source: §1 · p. 2

2 Related Work

• Table 1 compares chemistry benchmarks by answer source and whether the task requires explanatory attribution.Its caption defines UG and Grad as undergraduate and graduate, respectively.Source: Table 1 · p. 2
• Chemistry benchmarks cover property prediction, structure elucidation, name conversion, reaction outcomes, and pathway generation, while literature-based QA mainly tests factual retrieval.The cited examples include ChemCoTBench, oMeBench, and SUPERChem for reaction outcome prediction or pathway generation.Source: §2 · p. 2
• General scientific benchmarks such as GPQA and SciBench include chemistry questions but do not involve molecular diagrams or reaction schemes.Source: §2 · p. 2

3 Method

• Four specialist agents acquire papers, process figures, ground text, and generate questions, while quality control checks outputs between stages.The provenance tree links accepted questions to their source paper, figure, structure, and supporting evidence.Source: §3 · p. 3

3.1 Visual Processing

• Visual processing isolates individual structures from chemistry figures and recovers identifiers that connect them to article text.The pipeline retains segmented structure images because SMILES conversion disproportionately removes complex structures, and it highlights each target in the original figure to recover its identifier.Source: §3.1 · pp. 3–4

3.2 Text Grounding

• Text grounding retrieves evidence using each structure’s identifier, image, caption, and figure references, expanding sentences with related context from the same paragraph.QC2 rejects entity mismatches and non-explanatory passages; partial explanations may remain as context, but a structure advances only when its evidence block contains an explanatory sentence.Source: §3.2 · p. 4

3.3 Question Generation

• Question generation uses the source paper’s feature–mechanism relation as the correct answer and pairs other visible features with plausible mechanisms as distractors.The question stem uses the effect alone; same-effect competition, matched confidence, and randomized answer positions reduce textual shortcuts.Source: §3.3 · p. 4

4.3 Quality Validation

• Experts assign a mean quality score of 4.53, and human–LLM agreement averages AC2 0.919 across the four dimensions.All questions pass the 4.0 expert-score threshold; the judge’s mean scores on the expert subset are close to those on the full accepted benchmark after Holm–Bonferroni correction.Source: Table 2 · p. 5, §4.3 · pp. 5–6

4.2 Dataset Statistics

• The benchmark contains 55.9% Mechanistic, 39.7% Design, and 4.3% Condition questions, alongside 338 molecule questions and 842 reaction questions.Each source paper contributes 3.08 questions on average, with a median of 3 and a maximum of 8; automated construction supports extension to new papers and subdomains without manual question writing.Source: §4.2 · p. 5

5.1 Main Results

• 78.6% accuracy for GPT-5.4 led full-benchmark results, followed by Claude Sonnet 4.6 at 78.2% and Gemini-3.1-Pro at 78.0%, with no significant pairwise differences.Qwen3-VL-30B reached 58.8%; experts averaged 73.3% on the 600-question subset.Source: Table 3 · p. 6, §5.1 · p. 6

5.2 Visual Necessity

• 53.5–54.0-point accuracy drops followed image removal, from 74.0–78.5% with images to 20.5–24.5% without them, below the 25% random baseline.The question and four options remained unchanged in the text-only control.Source: Table 4 · p. 7, §5.2 · p. 7

5.3 VLM Error Analysis

• 48% of incorrect responses across three models were attribution errors; for GPT-5.4, attribution accounted for 100 of 140 errors (71.4%).Three experts agreed on error types in 96.5% of cases, with Gwet’s AC2=0.984.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4 Agentic Construction Pipeline Analysis

• 68.5% of retained structures were discarded by SMILES-first conversion, which preserved only 2 of 43 explanatory sentences in the sampled analysis.The reported extraction failures were common for large rings, densely substituted structures, and nonstandard drawings.Source: §5.4 · p. 8

Identifier-based retrieval enables precise text grounding

• Removing VLM-based identifier extraction lowered entity match from 76% to 50%, while replacing it with MolCoref lowered the rate to 35%.MolCoref often supplied no identifier or one for a neighboring compound, misleading downstream retrieval.Source: Identifier-based retrieval enables… · p. 8
• Context expansion increased explanatory sentences found in 100 sampled papers from 26 to 43, a 65% gain, but only 13.1% of structures with retrieved text had explanatory evidence in the full corpus.An expert audit found a comparable rate: 13.4% of structures with retrieved text contained an explanatory sentence.Source: Identifier-based retrieval enables… · p. 8

• Removing the image lowered accuracy to 20.5–24.5%, while at least five of eight experts selected the labeled answer for 94.0% of questions without seeing the source paper.Together, these findings support the benchmark’s visual requirement and expert-consensus grounding.Source: §6 · p. 8
• Attribution errors fell 38% from ChemVLM to GPT-5.4, versus 80–90% for perception and knowledge errors, and Evidence-Grounded Attribution improved three open-source models by 3.1–5.3 points.The authors suggest training models to compare competing explanations within a specific structural context.Source: §6 · p. 8

• ChemExplain is a benchmark of 1,180 chemistry questions constructed automatically from peer-reviewed papers and assessed by eight chemistry experts.Source: §7 · p. 8
• 78.6% accuracy was reached by the strongest evaluated model, and 71.4% of its errors were classified as attribution failures.Source: §7 · p. 8
• The authors identify explanatory attribution as an unresolved problem in multimodal chemical reasoning and ChemExplain as a way to measure progress.Source: §7 · p. 8

• ChemExplain uses multiple-choice questions, while free-form explanation generation remains a complementary setting.Source: Limitations · p. 9
• The benchmark focuses on 2D structure drawings rather than spectra, 3D conformations, or crystal structures.Source: Limitations · p. 9
• Retained source evidence could serve as a reference for future free-form explanation evaluation.Source: Limitations · p. 9

Ethical Considerations

• Source papers carry Creative Commons licenses that permit text mining and derivative works, while each benchmark item retains provenance to its source paper.Users considering commercial applications should consult the license of each source article.Source: Ethical Considerations · p. 9
• ChemExplain is released under CC-BY-NC-SA 4.0 for research purposes.The authors advise users intending commercial applications to consult each source article's license.Source: Ethical Considerations · p. 9
• The authors state that they will release the construction code and full prompt templates upon publication.Source: Ethical Considerations · p. 9

A Extended Related Work

• ChemExplain's related-work distinction is that prior chemistry benchmarks mainly test prediction, retrieval, or extraction rather than attribution of effects to visible structural features.The cited extraction pipelines retrieve associated text but do not validate whether it explains a structure-to-effect relation.Source: A Extended Related Work · pp. 12–13

B Representative Benchmark Example

• Figure 6 illustrates a question about durability and reaction quantum yield, with the correct explanation grounded in source-paper evidence and distractors tied to other visible features.Source: B Representative Benchmark Example · p. 13

C.1 Visual Processing

• Visual processing classifies figures, segments multi-molecule images and reaction schemes, extracts structure identifiers, and validates candidate regions as 2D chemical structures.MolCoref recovers identifiers for only about 27% of molecule figures and 15% of reaction schemes in the reported evaluation.Source: §C.1 · p. 13

C.2 Text Grounding

• Text grounding retrieves sentences by structure identifier and expands them with related sentences from the same paragraph when explanatory evidence spans adjacent sentences.The retrieval package includes the identifier, structure image, figure caption, and in-text figure references.Source: §C.2 · p. 13

C.3 Question Generation

• Question generation separates effect from feature and mechanism, keeps the source-paper mechanism as the correct explanation, and uses other visible features for distractors.Candidates are discarded if the explanatory text is too vague to supply all three fields; the question-generation call does not receive feature or mechanism fields.Source: §C.3 · p. 14

D.1 Human Expert Performance

• On 600 questions, at least five experts select the labeled answer for 94.0%, while experts who miss it choose different distractors in 77.5% of cases.GPT-5.4's 140 errors are concentrated on questions that most experts answer correctly, with attribution accounting for 71.4% of its errors overall.Source: §D.1 · p. 14

D.2 Has GPT-5.4 Closed the Attribution Gap?

• 94.1% versus 70.6% accuracy on within-domain questions contrasts with GPT-5.4's 84.6% versus Expert A's 42.3% out-of-domain.The authors conclude GPT-5.4 has not closed the attribution gap; its competitive overall accuracy reflects broader domain coverage rather than stronger explanatory attribution.Source: Table 10 · p. 15, §D.2 · p. 15

D.3 VLM Limitations Revealed by the Construction Framework

• Of 9,670 structures with retrieved text, only 1,264 (13.1%) contain explanatory sentences, and explanatory content can be missed when sentences are evaluated in isolation.On a 100-paper subset, context expansion recovers 40% of explanatory sentences missed by sentence-isolated evaluation.Source: §D.3 · pp. 15–16

D.4 Interventional Validation of the Error Taxonomy

• The interventions test error types on 440 distinct questions using expert-verified structural descriptions and option-matched chemical-knowledge statements.Each question is evaluated three times under four conditions and scored by majority vote.Source: §D.4 · p. 16
• Structural descriptions and chemical knowledge improve correction of perception errors and knowledge gaps, while no intervention significantly changes attribution-error correction.For GPT-5.4, structural descriptions raise perception-error correction from 20.0% to 80.0%, and chemical knowledge raises knowledge-gap correction from 16.7% to 70.0%.Source: §D.4 · p. 16, Table 11 · p. 17, Table 12 · p. 17

E.1 Agreement Metric

• The authors use Gwet's AC2 because the upper-clustered quality ratings create a prevalence paradox for chance-corrected agreement measures.They report a maximum human–LLM AC2 standard error of 0.022, corresponding to an approximate 95% confidence-interval halfwidth of 0.043.Source: §E.1 · p. 17

E.2 Sample Balance

• After Holm–Bonferroni correction, none of the four subset-versus-benchmark judge-score comparisons is statistically significant at the 5% level.The smallest p-value is 0.021, above the first corrected threshold of 0.0125.Source: Table 14 · p. 17, §E.2 · p. 17

E.3 Agreement Across Judge Models

• Three judges from different model families achieve average human–LLM AC2 values of 0.919, 0.938, and 0.949, respectively.The authors interpret the comparable agreement as reflecting the quality criteria rather than the judge choice.Source: §E.3 · pp. 17–18, Table 15 · p. 18

E.4 Ablation Replication Across Model Families

• With Claude-Sonnet-4-6, removing image-first construction, replacing VLM-based identifier extraction, and removing context expansion each degrade the corresponding metric.The replicated ablation reproduces the pattern reported in Table 5, although absolute counts shift with the model.Source: §E.4 · p. 18, Table 16 · p. 18

E.5 Rejected Question Analysis

• Structure Necessity accounts for 76.7% of rejected questions, while Correct Answer Unambiguity accounts for another 20.0%.Rejected questions often could be answered without the depicted structure or had another defensible option.Source: Table 17 · p. 18, §E.5 · p. 18

E.6–E.7 Error and Rejection Examples

• Figures 10–12 provide representative examples of attribution errors, perception errors, and knowledge gaps, while Figures 13–14 show two common rejection causes.Source: §E.6 · p. 18, §E.7 · p. 18

E.6 VLM Error Case Studies

• The error case studies distinguish attribution failures from perception errors and knowledge gaps.The supplied examples are Figures 10, 11, and 12, respectively.Source: §E.6 · p. 18

E.8 Accuracy Uncertainty Across Source Papers

• Paper-level resampling finds no statistically distinguishable paired accuracy differences among GPT-5.4, Claude Sonnet 4.6, and Gemini-3.1-Pro.Each model's paired-difference 95% confidence intervals include zero, and keeping one question per paper changes accuracy by at most 0.3 points per model.Source: Table 18 · p. 19, §E.8 · p. 19

E.9 Accuracy by Reasoning Type

• Design questions are the easiest category for evaluated models and human experts, while Mechanistic and Condition questions are more challenging.For Qwen3-VL-30B, Mechanistic and Condition accuracy is 54.7% and 54.9%, versus 65.0% on Design.Source: §E.9 · p. 19

G SMILES-First Pipeline

• The evaluated SMILES-first alternative converts segmented structures to strings and discards a reaction if extraction fails for any component.Extracted strings undergo round-trip validation using redrawing, graph comparison, and a similarity threshold of 0.85.Source: G SMILES-First Pipeline · p. 20

F Evidence-Grounded Attribution

• Evidence-Grounded Attribution improves the three evaluated open-source and chemistry-specific models by 3.1–5.3 percentage points, with a strongest result of 55.6%.The method grounds each candidate explanation in the depicted structure before comparing its mechanism; gains remain moderate because it targets attribution rather than perception and knowledge failures.Source: Table 19 · p. 20, F Evidence-Grounded Attribution · p. 20

H Prompt Templates

• The prompt-template appendix presents the explanatory-decomposition, question-and-distractor generation, and LLM-judge screening prompts.Source: H Prompt Templates · p. 20

I Human Evaluation Platform

• Experts answer without source text, then review paper evidence and score four quality dimensions on a 1–5 Likert scale.The platform presents these as separate blind-answer and evidence-review phases.Source: I Human Evaluation Platform · pp. 20–21

J Benchmark Question Examples

• Examples cover Mechanistic, Design, and Condition questions for both molecule and reaction entries, showing the image, options, question, and correct answer.The benchmark composition is 55.9% Mechanistic, 39.7% Design, and 4.3% Condition.Source: J Benchmark Question Examples · p. 21, Figure 5 · p. 21

K Design Considerations and Future Work

• 68.5% and 68.0% accuracy on pre- and post-cutoff question sets show no measurable GPT-4o accuracy increase from prior exposure to source papers.The comparison used two category-matched sets of 200 questions; the reported difference was one question (95% CI [−8.6, 9.6] points).Source: K Design Considerations and Future Work · p. 21
• The generator rewrites source-paper-supported feature–mechanism relations, and GPT-5.4, Claude Sonnet 4.6, and Gemini-3.1-Pro score within 0.6 points of one another.Without images, the three frontier models reach similar accuracies of 20.5–24.5%.Source: K Design Considerations and Future Work · p. 21
• Independent experts select the labeled primary explanation by majority on 94.0% of the expert subset, while ambiguity screening removes questions with another equally defensible option.Condition questions make up 4.3% (51 questions) because source papers rarely explicitly explain why a feature requires particular reaction conditions; the corpus is drawn from the PubMed Central Open Access Subset and journals with SJR ≥2.0.Source: K Design Considerations and Future Work · pp. 21–22
• The error examples distinguish wrong feature attribution, missed or misidentified structure, and missing chemical knowledge; rejection examples illustrate unnecessary structure analysis and ambiguous answers.The paper also suggests future versions could retain jointly explainable cases through graded or multi-answer evaluation.Source: K Design Considerations and Future Work · p. 22, Figure 10 · p. 25, Figure 11 · p. 26