Source-linked AI summary
ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation
Eric Inae, Tim Gunn, Chris Bond, Meng Jiang
TL;DR
Existing chemistry benchmarks provide limited evidence about robust reasoning because they often narrow task coverage and overlook instruction and representation variation. ChemDIRT addresses this by evaluating 23 models across eight chemistry-reasoning categories under diversified tasks, prompts, representations, and modalities, finding uneven and highly sensitive performance. The benchmark’s scope is limited to representative molecular-level tasks and does not cover several laboratory, spectroscopy, materials, or long-horizon experimental settings.
Problem
Existing chemistry benchmarks often cover narrow task sets and fixed formulations, limiting evidence about robustness to instruction and molecular-representation variation.
Method
ChemDIRT jointly varies tasks, instructions, molecular representations, and text or image inputs across eight reasoning categories, with 4,484 examples evaluated on 23 models.
Results
Across 23 models, performance is uneven and often changes substantially with the task, prompt, or molecular representation, with specialized systems showing narrow strengths and weaknesses.
Takeaways & Limitations
ChemDIRT cautions against broad capability claims from a single task, prompt, or representation and supports auditable, reproducible comparison through released data, templates, and scoring code.
Takeaways & Limitations
ChemDIRT samples representative molecular-level tasks but excludes laboratory execution, spectroscopy, materials synthesis planning, and long-horizon experimental workflows; its image evaluation covers only molecular graphs.
Abstract
from arXiv · showhide
The rapid advancement of large language models (LLMs) has led to increasing interest in their application to scientific domains such as chemistry. However, existing chemistry benchmarks often provide only a narrow view of model capability, focusing on limited task sets while overlooking robustness to variations in problem formulation and chemical representation. As a result, reported performance may overestimate a model's true ability to reason consistently across realistic settings. To address this challenge, we introduce ChemDIRT (Diversified Instruction, Representation, and Task Benchmark), a comprehensive evaluation framework designed to assess the robustness of chemical reasoning in LLMs. ChemDIRT systematically measures model performance across variations in instructions and molecular representations while spanning eight categories of chemistry tasks. By evaluating both accuracy and consistency under these controlled perturbations, ChemDIRT provides a more reliable assessment of model reasoning capabilities than conventional single-format benchmarks. We benchmark a diverse set of open- and closed-source LLMs, revealing substantial prompt sensitivity, representation dependence, and uneven performance across task families.
I. INTRODUCTION
Chemistry benchmarks often cover limited task families and fixed formulations, leaving robustness to wording and molecular representation insufficiently measured. ChemDIRT addresses this gap by diversifying tasks, instructions, representations, and modalities in a broad evaluation framework.
- Semantically equivalent prompt wording can substantially shift ClinTox classification accuracy across models.Figure 1 compares four prompt variants asking the same question.
- Existing chemistry benchmarks cover only subsets of eight reasoning categories, so performance on one benchmark may not reflect broad chemical reasoning.The taxonomy includes description, classification, design, analytical, regression, computation, mechanistic, and relational reasoning.
- Molecular representations such as SMILES, SELF-IES, InChI, and IUPAC can produce different model behavior despite encoding the same molecule.Prior work reports low consistency across representations and task-dependent advantages for alternatives to SMILES.
- ChemDIRT contains 33 task–representation settings and 4,484 released examples across eight reasoning categories, evaluated with 23 open- and closed-source models.The benchmark treats instruction and representation diversity as design requirements because superficial choices can materially change measured performance.
- The benchmark jointly varies tasks, instructions, molecular representations, and text or image inputs while providing public data and evaluation code.These resources support reproducible scientific-model comparison.
II. RELATED WORK
Existing chemistry evaluations span recognition, qualitative inference, question answering, numerical prediction, computation, and relational reasoning, but are usually organized around narrow task families or representations. This motivates a common benchmark that evaluates diverse chemical reasoning settings together.
- Prior chemistry benchmarks assess molecular recognition and qualitative properties using representations such as SMILES, molecular graphs, and textual descriptions.Examples include identifying atoms, functional groups, rings, structural motifs, toxicity, bioactivity, and permeability.
- Chemistry question-answering and analysis benchmarks require applying conceptual knowledge across subfields and new contexts.They include chemistry subsets of general reasoning suites and domain-specific literature, hypothesis-generation, and molecular-property datasets.
- Regression and computational benchmarks evaluate continuous molecular properties and chemistry-grounded calculations such as stoichiometry and unit conversion.Representative regression datasets include ESOL, Lipo, FreeSolv, QM9, and Open Catalyst.
- Comparative and relational molecular reasoning remains relatively underrepresented, despite benchmarks for reagent ranking and molecule selection.
- Existing evaluations are typically designed in isolation around narrow task families or representations, motivating a benchmark that jointly spans diverse chemistry tasks.
III. THE CHEMDIRT BENCHMARK
ChemDIRT organizes molecular, reaction, and domain-knowledge reasoning into eight categories and evaluates them through 33 task settings. Its definitions are based on required reasoning, with explicit scope boundaries and standardized task construction.
- ChemDIRT defines eight categories: description, classification, design, analytical, regression, computation, mechanistic, and relational reasoning.
- The taxonomy categorizes tasks by the minimum chemical reasoning required for correctness rather than prompt wording or output format.Reformulation can change a category when the required reasoning changes.
- The benchmark covers reasoning over molecular entities, reactions, and chemical domain knowledge, excluding meta-level curation, distributional analysis, and uncertainty estimation.
- Table I lists 33 task settings and their evaluation counts, separating representation-specific variants when independently evaluated.
- 1) Description:: Description tasks infer structural attributes without empirical property inference, including carbon count, ring count, functional-group presence, and bond-type presence.Carbon and ring counts require integer answers; the other two tasks require binary labels, and all four use accuracy.
- 1) Description:: Description examples are sampled from QM9, ZINC, ChEMBL, and PubChem, with ground truth extracted using RDKit and alternative IUPAC names generated through the PubChem API.
2) Classification:
ChemDIRT treats classification as empirical molecular-property inference, evaluating binary labels on standard MoleculeNet datasets with accuracy.
- Classification tasks ask models to generate a binary label for a target empirical molecular property.
- The classification datasets are BACE, BBBP, ClinTox, and HIV, and performance is measured by accuracy.
3) Design:
Design tasks generate molecular structures from specifications, including representation and input-modality translation, while analytical tasks use open-form chemical reasoning and knowledge synthesis.
- Design tasks generate a molecular structure conditioned on a molecular specification.
- Design outputs are assessed with exact match and MACCS, Morgan, and RDK fingerprint similarity to measure strict and graded structural agreement.SMILES-to-IUPAC additionally separates structural exact match from literal IUPAC exact match.
- ChemDIRT evaluates IUPAC-to-SMILES, SMILES-to-IUPAC, image-to-SMILES, and image-to-IUPAC translation tasks.Image settings test robustness to both input modality and molecular representation.
- Analytical tasks require chemical knowledge synthesis and reasoning expressed through open-form language.They include literature-grounded question answering, true/false questions, and short-answer open responses.
5) Regression:
Regression tasks evaluate quantitative molecular property inference, while computation tasks evaluate numerical or symbolic answers across several chemistry settings.
- Regression tasks infer a numerical molecular property label from a prompt containing a molecule and target property.
- Regression evaluation uses log RMSE, MAE, relative RMSE, and RMSE, with lower values indicating better performance.The evaluated properties include ESOL, lipophilicity, hydration free energy, and O2 gas permeability.
- Computation tasks require numerical or coefficient-vector answers from physical-unit calculation, stoichiometric balancing, reaction-yield prediction, and molecular-weight calculation.Reference molecular weights are calculated with RDKit.
- Coefficient balancing is scored by accuracy, while the other computation settings use log RMSE, MAE, relative RMSE, or RMSE.
7) Mechanistic:
Mechanistic tasks reason over chemical transformations and reaction pathways through forward prediction and retrosynthesis across SMILES, molecular formulas, and graph images.
- Mechanistic tasks require reasoning over chemical transformations and reaction pathways.
- Forward prediction predicts a reaction’s major product, whereas retrosynthesis infers reactants from products and reagents.
- Each mechanistic task is evaluated with SMILES, molecular-formula, and molecular-graph-image inputs.Graph images are generated with RDKit.
- SMILES- and image-based outputs use exact match and MACCS, Morgan, and RDK fingerprint similarity, while formula-based outputs also measure atom cosine and atom-level F1.Formula-based settings use compositional metrics because formulas lack explicit bonding information.
8) Relational:
ChemDIRT evaluates relational reasoning over molecular entities using varied representations and prompt conditions. Its benchmark tables include relational tasks alongside description and molecular-weight tasks, while prompt variation is summarized across evaluated conditions.
- Reported results: Table IV panel (e) groups relational results with description and molecular-weight tasks.The table reports prompt-varied results as mean ± SD over evaluated conditions.
- Prompt variation: Figure 3 displays prompt variation using minimum–maximum bars, mean points, and model-specific colors.
- Relational task scope: Relational tasks reason over relationships between two or more molecular entities.The benchmark includes representation equivalence, chemical-similarity comparison, and molecular-weight comparison.
- Evaluation design: The relational subtasks use both SMILES and IUPAC inputs and are evaluated with accuracy.
B. Subset Selection
The benchmark controls evaluation cost by selecting representative subsets from oversized source test sets. Selection uses model-correctness patterns and swaps examples until subset accuracies closely match full-test accuracies within specified size bounds.
- Subset construction: Each example is represented by a binary vector indicating which reference models answer it correctly.
- Subset construction: K-means clustering selects an initial representative example nearest each cluster centroid.The desired subset size determines K.
- Agreement calibration: Examples are swapped to reduce deviations between subset and full-test accuracy across reference models.The procedure starts with the model showing the largest deviation and avoids increasing another model’s error beyond the previous maximum.
- Evaluation coverage: The evaluation also includes image-task results for vision-capable models and separate results for tasks without prompt variation.
- Stopping criterion: Every reference model must fall within one percentage point of its full-test accuracy.The smallest subset satisfying this criterion is selected, with a minimum of 100 and maximum of 500 examples.
D. Evaluation Setup
ChemDIRT evaluates 23 models across varied chemistry tasks, prompts, and molecular representations. Results show substantial instability, specialization, numerical unreliability, and representation-dependent changes in model ordering.
- Evaluation protocol: 23 models are evaluated zero-shot with a shared system prompt and task-specific output-format instructions.Vision settings include only six vision-capable models, with task-specific output restrictions and token limits.
- Cross-task results: Every model exhibits at least one major weakness, making performance on a single task family a poor proxy for general chemical reasoning.Strengths vary across translation, reaction prediction, regression, open-response QA, structure generation, and other task families.
- Prompt sensitivity: On ClinTox, Llama3 accuracy ranges from 0.095 to 0.905 across semantically equivalent prompts.Falcon ranges from 0.095 to 0.824, while GPT-5.6-Luna ranges from 0.236 to 0.899.
- Prompt sensitivity: 0.100–0.991 is Galactica’s atom-cosine range on formula-based retrosynthesis across prompts, and such gaps can reverse apparent model rankings.Mean and standard deviation provide a more representative summary than any single-prompt score.
- Model specialization: Specialized systems perform strongly on selected task families but remain weak on others.For example, ChemDFM models are strong on translation and mechanistic tasks but have low open-response exact match and true/false accuracy.
- Numerical reasoning: No model performs reliably across all regression and computation settings, with especially difficult oxygen-permeability and physical-unit tasks.Fluent chemical text or valid molecular strings do not imply reliable numerical reasoning.
- Representation dependence: Representation changes alter model ordering, including IUPAC gains over SMILES for some model–task pairs and a SMILES advantage for another.Performance also differs by translation direction and between forward reaction prediction and retrosynthesis.
V. DISCUSSION
ChemDIRT finds that current LLMs lack robust general chemical reasoning, with performance varying across prompts, tasks, and representations. Its scope remains limited to representative molecular-level tasks and finite, manually verified prompt templates.
- These findings caution against broad capability claims based on a single task or template because fixed evaluation choices can introduce substantial measurement bias.
- ChemDIRT samples representative molecular-level tasks rather than covering every chemistry domain, excluding several laboratory, spectroscopy, materials, and long-horizon workflow settings.
- Prompt robustness is measured with a finite set of manually verified templates, and subset evaluation cannot preserve every property of the complete source datasets.
- Current LLMs, including chemistry-specialized systems, do not demonstrate robust general chemical reasoning under zero-shot evaluation.
- Performance is uneven and often highly sensitive to prompt, task, or molecular representation.