Source-linked AI summary
A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images
Jennifer D'Souza, Fahad Ahmed, Cecilia Andrea Bustamante Andrade, Lina Frolova, Poorani Gnanasambandan, Dilshad Hussain, Muhammad Uzair Khan, Nkembeng Kevin Nkengfoa, Paul Praveen J., Fabio Priante, Sjoerd Franciscus van der Werf, Thomas Frederik Jan van Roeden
TL;DR
Scientific figures and tables remain difficult for digital libraries and multimodal AI systems to search and interpret reliably. This perspective uses ALD/E-ImageMiner and the ICDAR 2026 competition to frame future scientific-image challenges and proposes scientific conceptual understanding from images as a broader benchmark objective with an incremental roadmap.
Problem
Scientific figures and tables remain difficult to comprehensively index, retrieve, and analyze despite carrying primary experimental evidence.
Method
The paper examines ALD/E-ImageMiner’s complementary tasks and competition experience to define capabilities and future directions for scientific-image benchmarks.
Results
The paper proposes scientific conceptual understanding from images as a broader benchmark objective spanning contextual interpretation, claim support, and progressively richer scientific reasoning.
Takeaways & Limitations
Future benchmarks should incrementally broaden scientific-image coverage and evaluate contextual synthesis, hypothesis evaluation, provenance, uncertainty, and open-ended multimodal research.
Takeaways & Limitations
Existing vision-language models remain weaker on spatial reasoning, cross-modal synthesis, multi-step inference, and dense-figure numerical fidelity.
Abstract
from arXiv · showhide
Scientific figures and tables encode essential experimental evidence, yet remain difficult for digital libraries and multimodal AI systems to retrieve and interpret. The ALD/E-ImageMiner benchmark and ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching Scientific Figures provide 1,951 figures from 205 publications, expert-annotated for classification, data table extraction, summarization, and visual question answering. In these companion proceedings, we present a forward-looking perspective on how the benchmark can guide future scientific-image challenges. We examine how its tasks probe capabilities from visual and quantitative reading to domain-grounded reasoning and evidential justification, and how Bloom-informed question design can support deeper scientific understanding. We propose "scientific conceptual understanding from images" as a long-term benchmark objective, with future directions including broader domains and figure types, contextual and cross-document synthesis, hypothesis evaluation, provenance, uncertainty, counterfactual grounding, and open-ended multimodal research. This perspective connects the ICDAR 2026 challenge to a broader agenda for machine-actionable scientific visual knowledge and verifiable multimodal scientific AI.
1 Introduction
Scientific figures often serve as primary evidence in scientific communication, yet multimodal AI systems remain unreliable at interpreting their numerical, spatial, relational, and contextual content. The ALD/E-ImageMiner benchmark and ICDAR 2026 competition address this need while framing scientific conceptual understanding from images as a broader research agenda.
- Motivation: Scientific visual artifacts encode experimental observations, quantitative results, structural relationships, and methodological details, often constituting the primary record of evidence for scientific claims.The introduction emphasizes that figures and diagrams are not merely illustrations accompanying text.
- Problem: General-purpose vision–language models remain uneven on scientific figures because interpretation requires numerical fidelity, domain-specific notation, spatial and relational reasoning, cross-panel integration, and experimental context.Such models may misread axes and legends, overlook visual relations, or produce conclusions unsupported by the figure.
- Benchmark contribution: The ALD/E-ImageMiner benchmark and ICDAR 2026 competition evaluate figure classification, data table extraction, summarization, and visual question answering on expert-annotated ALD/E literature figures.The benchmark includes experimental and simulation-based atomic layer deposition and etching literature.
- Research agenda: The companion proceedings connect the current challenge to capabilities spanning visual localization, quantitative reading, domain-grounded interpretation, reasoning, and evidential justification.This contribution complements the competition report and participating-team studies by articulating a longer-term research agenda.
- Future directions: The proposed agenda introduces scientific conceptual understanding from images and calls for Bloom-informed questions plus future extensions involving richer representations, cross-document synthesis, hypothesis evaluation, provenance, and uncertainty.The scope extends across materials science and related engineering domains.
2 Related Work: Gaps in Scientific Image Understanding
Existing benchmarks advance chart extraction, classification, and visual question answering, but scientific figures demand domain-specific, multimodal, spatial, and multi-step reasoning. Digital-library systems likewise remain limited in grounding, faithful extraction, and joint evaluation across capabilities.
- Benchmark coverage: General chart benchmarks emphasize synthetic or programmatically generated visualizations with template-based questions and automatically generated plots.FigureQA, DVQA, and PlotQA represent this benchmark tradition.
- Benchmark coverage: Human-annotated materials-science benchmarks address real scatter plots containing heterogeneous axes, dense points, fitted curves, and categorical visual encodings.PolyCompChartIE and MetalThermoChartIE target chart-to-table extraction from polymer-composite and metal-thermophysical-property figures.
- Scientific reasoning demands: Scientific figures require reasoning over domain notation, experimental conditions, measurement techniques, spatial organization, and relationships across panels or modalities.These demands extend beyond chart classification, chart-to-table conversion, and visual question answering.
- Scientific reasoning demands: Evaluations show that strong general visual performance does not reliably transfer to scientific figures, especially for spatial reasoning, cross-modal synthesis, multi-step inference, table reasoning, formula recognition, and fine-grained analysis.Models may perform well on equipment recognition, direct numerical extraction, captioning, or salient-entity recognition while relying heavily on textual cues.
- Digital-library systems: Digital-library integration remains limited in domain-specific interpretation, panel-level grounding, numerically faithful extraction, and joint evaluation of multiple capabilities on the same figures.ORKGEx illustrates the need to combine language and vision models because conventional OCR cannot capture jointly expressed text, graphical marks, spatial relations, and domain-specific symbols.
3 ALD/E-ImageMiner as a Testbed for Scientific Visual Intelligence
ALD/E-ImageMiner is a testbed for progressively demanding scientific visual intelligence, spanning heterogeneous figure understanding, quantitative extraction, summarization, and domain-grounded visual question answering. Its Bloom-informed design supports deeper reasoning and motivates future evaluations of creation, cross-panel synthesis, provenance, and uncertainty.
- Benchmark scope: The benchmark contains 1,951 figures from 205 ALD/E publications across 49 figure categories, including charts, spectra, diagrams, process flows, and composite panels.It is presented as a testbed rather than merely a figure collection.
- Capability progression: Its four tasks operationalize a progression from identifying visual form to recovering structured quantitative data, summarizing scientific messages, and answering domain-grounded questions.Together, the tasks probe increasingly demanding vision-language capabilities in scientific settings.
- Bloom-informed VQA: Bloom-informed VQA uses process-oriented, comparative/trend, structure–property, and application/performance families to span remembering, understanding, applying, analyzing, and evaluating.Questions connect ALD/E processing, material structure and composition, measured behavior, and device-level or practical performance.
- Evidential justification: Explanatory paragraph answers evaluate whether conclusions are scientifically coherent and grounded in the figure, not merely whether they are correct.Answer formats also include yes/no, factoid, and list responses for different levels of precision and granularity.
- Future directions: Future editions could extend the benchmark toward scientifically plausible creation tasks, paired cognitive-depth questions, cross-panel reasoning, provenance-aware answers, and uncertainty assessment.These additions are proposed to enable richer evaluation of scientific visual reasoning.
4 Toward “Scientific Conceptual Understanding” from Images: A Challenge Roadmap
The roadmap proposes scientific conceptual understanding from images as a long-term objective requiring multimodal systems to connect visual evidence with experimental conditions, domain knowledge, alternative evidence, and warranted conclusions. It recommends expanding scientific coverage and developing increasingly contextual, comparative, uncertainty-aware, and open-ended tasks that support machine-actionable, verifiable research workflows.
- Long-term objective: Scientific conceptual understanding requires using visual evidence within a scientific process, not merely recognizing forms, extracting values, or generating plausible descriptions.Systems should identify observations, relate them to conditions and domain knowledge, compare alternative evidence, and determine warranted conclusions.
- Domain and figure expansion: Sci-ImageMiner should expand incrementally across materials science and related engineering disciplines while preserving expert annotation and domain grounding.Proposed extensions include thin films, semiconductor processing, two-dimensional materials, catalysis, electrochemistry, photovoltaics, polymers, ceramics, composites, chemical engineering, electrical engineering, and mechanical or aerospace engineering.
- Domain and figure expansion: Future challenges should prioritize the scientific functions of visual representations and use metadata to evaluate in-domain performance, neighboring-field transfer, and cross-convention generalization.Candidate representations include microscopy, diffraction, reciprocal-space, spectroscopic, reactor, process-flow, circuit, device, and timing figures.
- Contextual and comparative reasoning: Context-grounded, hypothesis-oriented, and cross-document tasks should require models to distinguish visible information from textual or domain-derived conclusions, compare evidence, test competing explanations, and revise assessments.Structured records can expose selected evidence, hypotheses, predicted observations, tests, resulting evidence, and revised conclusions for checking against figures and annotations.
- Counterfactual grounding, provenance, and uncertainty: Counterfactuals, provenance, and uncertainty should test whether model answers respond to changed scientific evidence rather than relying on explanation quality alone.Controlled interventions could remove panels, alter values, exchange labels, add contradictory evidence, or modify irrelevant formatting.
- Open-ended multimodal research challenges: The eventual benchmark should combine reproducible task diagnostics, structured epistemic evaluation, and expert assessment to measure responsible participation in evidence-driven, self-correcting science.The broader vision is a continuously expanding environment that retrieves and compares visual evidence, produces machine-actionable knowledge, proposes explanations, requests informative measurements, and reports remaining uncertainty.
5 Conclusion
The paper presents ALD/E-ImageMiner as a domain-grounded testbed for making experimental evidence in scientific figures and tables more searchable, comparable, and integrable. It proposes scientific conceptual understanding from images as a long-term objective and outlines incremental benchmark development toward broader, machine-actionable visual-evidence representations.
- 5 Conclusion: ALD/E-ImageMiner addresses the difficulty of searching, comparing, and integrating experimental evidence in scientific figures and tables through complementary benchmark tasks.The tasks cover classification, structured data extraction, summarization, and visual question answering.
- 5 Conclusion: The paper proposes scientific conceptual understanding from images as a long-term objective for multimodal scientific-image benchmarks.
- 5 Conclusion: Future Sci-ImageMiner development should incrementally broaden coverage before adding contextual interpretation, cross-figure synthesis, hypothesis evaluation, provenance, uncertainty, and open-ended multimodal research.The proposed progression begins within materials science and related engineering disciplines.
- 5 Conclusion: Such benchmarks could help digital libraries move beyond caption-based indexing toward machine-actionable representations of visual evidence.
Data availability statement
The ALD/E-ImageMiner benchmark is publicly released with images, annotations, metadata, structured files, evaluation scripts, and guidelines, but source PDFs are excluded. Its mixed-rights licensing requires users to consult source-publication terms before reuse, redistribution, or commercial exploitation.
- Data availability statement: The public release includes scientific figure images, machine-readable annotations and metadata, structured content files, evaluation scripts, and submission guidelines.Source article PDFs are intentionally excluded from the GitHub distribution.
- Data availability statement: The benchmark is a mixed-rights, non-commercial resource rather than a corpus covered by one blanket open license.Annotations and generated metadata use CC BY 4.0, while extracted images and source-derived content follow their source articles’ rights and reuse terms.
- Data availability statement: Users must consult corresponding source publications and rights-holder terms before redistributing, reusing, or commercially exploiting the corpus.The licensing obligations differ between annotations and generated metadata versus extracted figure images and source-derived content.
Underlying and related material
The ALD/E-ImageMiner project provides benchmark access, documentation, and related resources, including submission guidelines and official evaluation scripts. Codabench competition pages host the four task leaderboards, which remain open indefinitely while the platform remains available.
- The ALD/E-ImageMiner competition project page provides access to the benchmark dataset, documentation, and related resources.
- Submission-format guidelines and official competition evaluation scripts are available through the project’s GitHub repository.
- Codabench hosts separate competition pages for classification, data table extraction, summarization, and visual question answering.
- The live Codabench leaderboards remain open indefinitely for submissions, subject to continued availability of the Codabench platform.
Funding
The ALD/E-ImageMiner benchmark dataset was created with support from the NFDI4 DataScience initiative, funded by the German Research Foundation.
- The dataset creation was supported by NFDI4 DataScience under German Research Foundation funding (DFG Grant ID: 460234259).