PaperlayerView exact revision on arXiv

Source-linked AI summary

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark

Hanyao Guo, Mo Cui, Zihan Tan, Leixin Sun, Yichun Wang, Yinan Shen, Guojun Zhu, Jia Jiang, Erhao Chen, Zui Chen, Wenzhe Gao, Zhihao Yu, Qijun Wang, Yetian Zhu, Jiande Chen, Chuhang Pei, Yao Jiang, Yichen Fan, Pinhan Wang, Xiaorui Pan, Qianyi Zhu, Yang Liu, Mang Ye, Guancheng Wan

arXiv:2610.20012v1cs.LG

TL;DR

Existing chemistry benchmarks score answer endpoints, leaving reasoning steps, modality differences, and planning versus execution difficult to distinguish. ChemArgus combines rubric-point partial credit, controlled image-text evaluation, and protocols that share a scoring unit while varying context. Across the reported runs, whole-problem solving is the lowest-scoring metric in every run, while scores rise toward step-level evaluation.

  • Problem

    Endpoint-scored chemistry benchmarks cannot distinguish factual recall from derivation, controlled visual from textual input, or global planning failures from local execution errors.

    Source: §1 · p. 3, §2.3 · p. 4

  • Method

    ChemArgus grades subquestions point by point with partial credit, pairs real images with controlled text descriptions, and compares protocols that share a scoring unit but vary context.

    Source: §1 · pp. 2–3

  • Results

    Whole-problem solving is the lowest metric in all twenty-five reported runs, and scores climb from whole problem to step-level evaluation across them.

    Source: §5.1 · p. 8, §6 · p. 13

  • Takeaways & Limitations

    The three shared-unit protocols make score gaps diagnostic of integration, propagation, or local reasoning within the evaluated chemistry tasks.

    Source: §6 · p. 13

Abstract

from arXiv · show

Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still score answer endpoints alone. Such scoring conflates memorized facts with stepwise derivation, treats visual and textual inputs as interchangeable, and confounds long-horizon planning with stepwise execution. These blind spots prevent any diagnosis of where the reasoning chain breaks in a model. We present ChemArgus, a competition-level chemistry benchmark drawn from chemistry competition material, textbooks, and research papers, whose scoring replaces the answer endpoint with three measurement properties. Fine-grained scoring grades every subquestion point by point with partial credit, so a score lands at the level of the step rather than the outcome. Multimodal capability is priced on every problem containing a real image, paired with a controlled text description, so visual understanding is measured against text on identical tasks. Diagnostic attribution reads paired metrics that share one scoring unit and differ only in context construction, so a gap between two scores isolates a single cause, namely integration, propagation, or local reasoning. GPT-5.6-Sol scores 23.2% on unassisted whole-problem solving against 28.8% and 30.4% under the two sequential metrics, a gap measuring the integration burden. Through rubric partial credit and these cause-isolating score gaps, ChemArgus shows not only whether a model answers chemistry correctly but where its reasoning chain fails, so that every score it reports doubles as a diagnosis, a readout of the chain rather than a verdict, aiming to guide language models step by step on their path toward chemistry expertise, one corrected step at a time, each step priced by the rubric, point by point.

1. Introduction

Existing chemistry benchmarks score final answers, leaving recall versus derivation, visual versus textual input, and global planning versus local execution unresolved. ChemArgus addresses these gaps with rubric-level partial credit, controlled image-text comparisons, and protocols that isolate causes of score differences.

1. Introduction

  • Endpoint scoring conflates factual recall with derivation, treats modalities as interchangeable, and cannot distinguish failed planning from a local execution error.The introduction identifies these as gaps in existing evaluation and motivates a fine-grained, multimodal diagnostic instrument.Source: §1 · pp. 2–3
  • ChemArgus is a high-difficulty chemistry benchmark for open-ended reasoning and multimodal understanding, built from chemistry competition material, textbooks, and research literature.Problems are designed for chemical correctness, derivability from the statement, and discrimination between factual recall and genuine reasoning.Source: §1 · pp. 2–3
  • Every subquestion receives point-by-point rubric scoring with partial credit, placing scores at the reasoning-step level rather than only at the final outcome.Every problem pairs real images with controlled text descriptions, enabling visual understanding to be compared against text on identical tasks.Source: §1 · p. 3
  • Three protocols share one scoring unit and vary context construction, so score gaps isolate integration, propagation, or local-reasoning differences.Full problem presents the whole task at once; sequential carry retains prior model answers; oracle scaffolded supplies standard intermediate results.Source: §1 · p. 3

2. Related Work

Prior work established difficult open-ended chemistry evaluation, multimodal chemistry tasks, and rubric-based or multi-step scoring. The reviewed benchmarks still lack a combined controlled design that localizes reasoning failures and measures modality against controlled text.

2.1. Chemistry Knowledge and Reasoning Benchmarks

  • Existing chemistry reasoning benchmarks expanded from factual and multiple-choice probes to expert open-ended tasks, but their endpoint scores do not locate where a wrong answer became wrong.Examples include open-ended, multi-level, quantitative, and multimodal chemistry benchmarks.Source: §2.1 · p. 3

2.2. Multimodal Chemistry Evaluation

  • Multimodal chemistry studies show figure-dependent failures, but none of the reviewed work pairs every figure with controlled text to separate image-conditioned scores from text content.The literature includes chemical-table understanding, multimodal chemistry tasks, and figure-reading benchmarks.Source: §2.2 · p. 4

2.3. Scoring and Protocols

  • Rubric scoring, deterministic checks, and multi-step evaluation have each been demonstrated, yet reviewed work does not jointly control grading artifacts and vary protocol while holding the scoring unit fixed.The paper identifies the remaining gap as a control surface for scoring and protocol comparisons.Source: §2.3 · p. 4
  • ChemArgus targets the combined gap with rubric-localized partial credit, controlled modality contrast, and protocol separation on one scoring unit.It also uses real images throughout and builds access control into its measurement design.Source: §2.3 · p. 4

3. Benchmark Design

ChemArgus structures expert chemistry problems as shared-context subquestion chains and evaluates them through three protocols that retain a common scoring unit. Its construction and dual-track grading support point-level diagnosis while keeping model-visible media separate from grading materials.

3.1. Task Formulation and Data Protocol

  • Each problem is a normalized JSON unit with a shared header and ordered subquestions, each carrying text, images, full marks, rubric points, and stable identifiers.Model-visible images have curated text descriptions, while rubric-private images and scoring materials are excluded from all model inputs.Source: §3.1 · p. 5

3.2. Dataset Construction

  • Problems are constructed from competition knowledge, textbooks, and research literature, then filtered through expert review, adversarial model probing, and student pilots.Final structured units retain model-visible images and descriptions, rubrics, and rubric-private images.Source: §3.2 · pp. 5–6

3. Benchmark Design

  • ChemArgus is designed as a measurement instrument, combining real image input, controlled image-text contrast, rubric-point partial credit, subquestion chains, and protocol separation.The paper contrasts these design properties with prior benchmarks in Table 1.Source: §3 · p. 5, Table 1 · p. 6

3.3. Three Evaluation Protocols

  • Full problem, sequential carry, and oracle scaffolded use the same subquestions and scoring unit but vary the context available when answering.Their score gaps distinguish propagation, integration burden under propagation, and combined integration-plus-propagation load.Source: §3.3 · pp. 6–7

3.4. Rubric-Grounded Scoring

  • Rubric points are graded independently with partial credit; formal chemistry answers receive deterministic equivalence checks, while flexible explanations receive judge verdicts with evidence and confidence.Low-confidence and boundary cases go to human review, and grading interactions are archived for traceability.Source: §3.4 · p. 7

4. Evaluation

The evaluation applies three protocols and three input modes to nine frontier models, with normalized rubric scores assigned per subquestion. This design enables comparisons of image input, reader assistance, and context construction.

4.1. Models

  • Nine frontier models are evaluated off the shelf, with default settings and no task-specific fine-tuning.Runs use zero prompting, with no few-shot examples, reasoning instructions, tools, or web search.Source: §4.1 · p. 7, §4.2 · p. 7

4.2. Scoring

  • Scores are normalized rubric points earned and are graded per subquestion by an independent LLM judge under an audited dual-track pipeline.Formal answers receive deterministic chemistry-equivalence checks, while flexible answers are judged against rubric points; human spot-checks report high agreement.Source: §4.2 · p. 7

4.3. Input Modes

  • The three input modes are real multimodal image blocks, text descriptions without images, and text paired with a Qwen3-VL reader the model invokes autonomously.The three metrics are full problem, sequential carry, and oracle scaffolded; their shared scoring unit makes score gaps diagnostic.Source: §4.3 · p. 8

5. Results

Across the reported runs, whole-problem scores are lower than sequential scores, while model rankings and modality effects vary by input mode. The results also show that adding an external reader can help or hurt depending on the model.

5.1. Overall Results

  • Claude Opus 5 leads every metric, while the top model varies by input mode and metric.Claude Opus 5 scores 53.8 on whole-problem solving and 64.9 under oracle scaffolded multimodal input; GLM-5.2 leads the reader panel on whole-problem solving at 52.0.Source: §5.1 · p. 8, Table 2 · p. 9
  • The whole-problem metric is lowest in all 25 runs, but the integration-burden gap varies by model and input mode.The gap ranges from 0.2 percentage points for Claude Opus 5 on text to 13.2 points for Gemini 3.8 Flash on text.Source: §5.1 · p. 8

5.2. Modality Contrast

  • On whole-problem solving, real images outperform text for Gemini 3.8 Flash, Grok 4.6, and MiniMax M3, while text leads for four other models.The image advantages are 2.0, 0.4, and 1.1 percentage points, respectively; the text advantages range from 6.5 to 7.7 points.Source: §5.2 · p. 9, Table 3 · p. 11
  • Modality advantages vary across metrics: Gemini 3.8 Flash's image advantage peaks on whole-problem solving, while Claude Opus 5's text advantage reverses at the step.Gemini's image advantage is 2.0 points on the whole problem; Claude's text advantage shifts from 7.7 points on the whole problem to a 3.2-point step advantage for images.Source: §5.2 · p. 9, Figure 5 · p. 11

5.3. Reader Effect

  • The reader changes scores in both directions: 19 of 27 metric cells decline, while GLM-5.2 gains 28.5 points on whole-problem solving.Claude Opus 5 falls 15.4 points on that metric; the pairing uses a Qwen3-VL reader that converts images to text.Source: §5.3 · p. 10, Table 4 · p. 12

5.4. Answers to the Two Questions

  • Across the 25 runs, whole-problem solving is the weakest stage; modality and reader effects are model-dependent rather than uniform.The whole-problem gap ranges from 0.1 points in flattened reader runs to 13.2 points for Gemini 3.8 Flash from text; reader effects range from a 28.5-point gain for GLM-5.2 to a 15.4-point loss for Claude Opus 5.Source: §5.4 · pp. 12–13

6. Conclusion

ChemArgus makes chemistry scores diagnostic by grading rubric points across protocols that separate integration, propagation, and local reasoning. Across nine frontier models, whole-problem integration is the largest measured cost, while modality and reader effects vary by model.

6. Conclusion

  • 23.2 vs. 28.8 and 30.4: GPT-5.6-Sol with images scores lower on whole problem than on either sequential metric.Gemini 3.8 Flash with text scores 46.5 on whole problem versus 59.0 and 59.7 on the sequential metrics.Source: §6 · p. 13
  • 2.0, 0.4, and 1.1 pp: Gemini 3.8 Flash, Grok 4.6, and MiniMax M3 gain from real images, while four other models lose 6.5 to 7.7 pp.The models losing are GPT-5.6-Sol, Qwen3.8-Max, Kimi K3, and Claude Opus 5.Source: §6 · p. 13
  • 28.5 pp: the reader effect lifts GLM-5.2 on whole problem but lowers Claude Opus 5 by up to 15.4 pp, a 43.8 pp roster swing.Source: §6 · p. 13
  • ChemArgus uses three protocols to measure integration, sequential error propagation, and local reasoning, with rubric partial credit and separate grading procedures for formal and flexible answers.Images shown to a model are isolated from the visual content used to grade it.Source: §6 · p. 13
  • 64.9: Claude Opus 5’s best reported score, under oracle scaffolded with images, still leaves more than a third of rubric credit unclaimed.Source: §6 · p. 13

Related papers

Loading 2610.20012v1…