PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlocking the Visual Record of Materials Science: A Large-Scale Multimodal Dataset from Scientific Literature

Subham Ghosh, Shubham Tiwari, Mohammad Ibrahim, Abhishek Tewari

arXiv:2606.29667v1cs.CVcond-mat.mtrl-scics.AI

TL;DR

Materials science figures contain valuable experimental knowledge, but compound figures and caption-only supervision hinder reliable panel-level image-text alignment. MatMMExtract addresses this by decomposing figures and generating taxonomy-guided annotations, producing a large panel-level dataset with strong detection and annotation benchmarks.

  • Problem

    Materials science databases omit much experimental context, while compound figures and caption-only supervision lack panel grounding, consistent visualisation taxonomies, and scientific summaries.

  • Method

    MatMMExtract parses open-access articles, detects and decomposes compound figures, and generates taxonomy-guided panel annotations for MatSciFig using language models.

  • Results

    391,606 annotated panels were produced; Gemini 3.1 Flash Lite had a 4.8% hallucination rate, and YOLO12-m achieved mAP_50 of 0.9227.

  • Takeaways & Limitations

    MatSciFig is the largest panel-level multimodal dataset in materials science, combining grounded sub-captions, visualisation categories, sub-categories, and concise summaries.

  • Takeaways & Limitations

    MatSciFig covers only Elsevier and Springer open-access articles, and its text-only annotations depend on the completeness and clarity of figure captions and reference sentences.

Abstract

from arXiv · show

The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.

1 Introduction

Materials science databases and existing multimodal resources omit much experimental and visual context, while caption-only supervision lacks panel grounding, taxonomy, and scientific-significance summaries. MatMMExtract addresses these deficits by producing the open MatSciFig panel-level resource and associated detection, annotation, and retrieval contributions.

  • Materials databases largely record tabulated property values while omitting experimental context, limiting their coverage of materials knowledge.
  • Existing multimodal efforts remain limited: Exsclaim focuses on microscopy, while MatCha, MatQnA, and MATRIX provide benchmarks rather than a large-scale, modality-diverse training corpus.
  • Caption-only supervision lacks panel grounding, a consistent visualisation taxonomy, and summaries of scientific significance needed for context-aware materials vision-language understanding.
  • MatMMExtract is an open-source pipeline that parses publisher XML, decomposes compound figures into panels, and generates grounded sub-captions, category/subtype labels, and summaries.
  • 2,811 manually annotated figures and 12,906 boxes comprise MaterialScope, supporting detector benchmarking across diverse materials domains and showing that domain-specific training drives accuracy.
  • 391,606 annotated panels from 180,571 figures comprise MatSciFig, with a 19-category taxonomy, six low-cost LLMs benchmarked for annotation reliability, and a CLIP-style dual-encoder baseline for downstream research.

2 Methods · 2.1 Source Data Collection

MatMMExtract collects open-access materials science articles and extracts figures, captions, and in-text references before detecting compound-figure panels and generating taxonomy-guided annotations. The source corpus is restricted to redistributable Creative Commons licenses, with 62% of curated figures identified as compound.

  • 2 Methods: MatMMExtract follows three stages: source XML extraction, sub-panel detection and localisation, and LLM-based annotation guided by a materials science taxonomy.The LLM uses captions, reference sentences, and the predefined taxonomy to generate structured annotations.
  • 2.1 Source Data Collection: The corpus is assembled from OpenAlex and Scopus, combining fine-grained and coarse open-access metadata categories.OpenAlex supports specific Creative Commons filtering, whereas Scopus labels open-access articles as gold or hybrid.
  • 2.1 Source Data Collection: Figures, captions, and in-text reference sentences are extracted from Elsevier and Springer article XML files retrieved through publisher APIs.The pipeline parses structured XML and downloads the linked figure images.
  • 2.1 Source Data Collection: The collection is restricted to CC-BY and CC-BY-NC articles to ensure compatibility with downstream redistribution.These licenses are selected because some other open-access categories may restrict derivative works.
  • 2.1 Source Data Collection: 62% of figures in the curated corpus are compound figures, complicating direct image-text alignment because one caption can describe multiple sub-images.This prevalence is consistent with the approximately 40 - 60% compound-figure rate reported for open-access biomedical literature.
  • 2 Methods: A compound figure detector decomposes figures into individual sub-panels to address the mismatch between compound images and captions.The detector identifies and localises sub-panels before downstream annotation.

2.2 Compound Figure Detection

MaterialScope addresses the domain gap in compound-figure detection through 2,811 manually annotated materials-science figures and enables a benchmark of six detector architectures. YOLO12-m achieves the strongest accuracy–compactness balance and is selected for the annotation pipeline.

  • Dataset construction: 2,811 materials-science figures were manually annotated across alloys, composites, ceramics, steels, polymers, and thin films, yielding 12,906 valid sub-panel bounding-box annotations.Two PhD-level researchers independently labelled sub-panel bounding boxes using Label Studio.
  • Dataset construction: MaterialScope uses 2,249 training, 281 validation, and 281 test images, with splits stratified to preserve sub-panel-label distributions.Agreement was assessed on a shared subset of 102 images using Cohen’s Kappa over complete panel-label sets.
  • Evaluation protocol: All models use a unified test protocol with confidence threshold 0.55, NMS IoU threshold 0.45, 1024 × 1024 inference, and IoU 0.50 matching.The evaluation reports precision, recall, F1, frequency-stratified AP, and throughput.
  • Detector comparison: 0.9227 mAP50 makes YOLO12-m the best retrained model, ahead of YOLO8-m at 0.9028, while using 20.15 M parameters and 86.82 GFLOPs.The F1 scores of the top YOLO variants are close, so mAP50 determines model selection.
  • Detector comparison: 0.6569 mAP50 makes DAB-DETR the weakest model, alongside the largest parameter count at 43.65 M and slowest throughput at 9.18 img/s.Its precision is 0.9525 but recall is 0.8150, whereas YOLO8-m reaches 14.31 img/s and YOLO12-m is adopted for balanced performance.

2.3 Caption, Category, and Summary Generation

After panel localisation, an LLM generates structured annotations from the figure caption and in-text reference sentence without receiving the panel image. A two-level materials-science taxonomy and constrained output format standardise categories, subtypes, sub-captions, and summaries.

  • Annotation generation: The second pipeline stage gives the LLM only the original figure caption and in-text reference sentence for each localised panel, never the panel image.Each atomic image unit receives structured annotation outputs grounded in these textual inputs.
  • Taxonomy: The taxonomy defines 19 broad visualisation categories with domain-specific subtypes covering experimental, computational, quantitative, and illustrative materials-science figures.Examples include Microscopy, Diffraction, Spectroscopy, Electrochemistry, Simulation, Schematic/Diagram, Photograph, and Generic Plot.
  • Output constraints: Category and subtype selection is constrained through a readable two-level taxonomy prompt and an API-level JSON schema requiring fields, strict category enumeration, and no extra properties.The subtype is constrained by the prompt, while other is reserved as a fallback when no defined subtype applies.
  • Grounding and consistency: Prompts require sub-captions to stay grounded in supplied text, selectively scope reference sentences to panels, vary sentence openings, and cap summaries.Group-level references are shared across panels, whereas panel-specific references apply only to their own sub-panel.
  • Annotation record: Each panel record contains its identifier, visualisation category, visualisation subtype, sub-caption, and summary alongside the cropped sub-panel image.These fields form the final annotation triplet with the image.

2.4 Benchmarking

Benchmarking shows Gemini 3.5 Flash leads classification and generation quality, while Gemini 3.1 Flash Lite offers comparable generation quality with lower hallucination and cost, motivating its downstream selection. Fine-grained plot distinctions and photographic categories remain difficult, especially for smaller models.

  • Category classification: 99.0% on Microscopy, 97.8% on Mechanical Test, and 95.5% on Thermal Analysis make Gemini 3.5 Flash the strongest overall category classifier.Gemini 3.1 Flash Lite is close behind, with 94.4% on Simulation and 92.6% on Schematic/Diagram.
  • Failure modes: 43.2% for GPT-5.4 Nano on Photograph makes it the weakest model on the hardest category, while GPT-5.4 Mini and Nano reach 76.8% and 55.6% on weak-point Simulation classification.Photographic content is often identifiable only from the image, whereas captions provide little disambiguation.
  • Sub-category classification: 100% on Fatigue/S-N Curve for Gemini 3.5 Flash contrasts with 11.5% for GPT-5.4 Nano, highlighting failures on fine-grained plot distinctions.GPT-5.4 Mini reaches 42.3% on Fatigue/S-N Curve, while GPT-5.4 Nano reaches 33.3% on Scatter Plot and 43.8% on Bar Chart.
  • Generation quality: 88.3% [83.4, 91.9] of Gemini 3.5 Flash sub-captions and 87.4% [82.4, 91.1] of summaries were rated good, with a 9.0% [5.9, 13.5] hallucination rate.Gemini 3.1 Flash Lite was statistically comparable, achieving 82.0% [76.2, 86.7] good sub-captions and 83.5% [77.8, 87.9] good summaries.
  • Model selection: 82.0% [76.2, 86.7] good sub-captions and 83.5% [77.8, 87.9] good summaries, combined with lower hallucination and fractional cost, led to selecting Gemini 3.1 Flash Lite for downstream generation.The four other models had caption good-rates of 56.8 to 58.8%, summary good-rates of 49.5 to 53.1%, and hallucination rates above the 10% threshold.

3 The MatmmExtract Package

MatMMExtract is released as an open-source Python package implementing the complete pipeline through six modules.

  • 3 The MatmmExtract Package: The open-source matmmextract package implements the complete pipeline as six modules corresponding to its described stages.It requires Python ≥3.10 and is installable from PyPI via pip install matmmextract.

4 Dataset

MatSciFig is a large-scale, panel-level materials science figure dataset created by decomposing compound figures and attaching structured annotations. It contains 391,606 panels from 180,571 figures, with broad visualization coverage and high annotation alignment quality.

  • Dataset construction: Compound figures are decomposed into atomic panels, which are paired with structured annotations generated from open-access alloy, composite, and ceramic literature.The annotations include panel-specific sub-captions, visualization categories, sub-categories, and extended summaries.
  • Dataset construction: MatSciFig contains 391,606 annotated panels extracted from 180,571 source figures, providing large-scale panel-level vision-language data for materials science.Each panel includes a panel-specific sub-caption, visualization category and subtype, and an extended summary derived from the source paper.
  • Dataset construction: 83.5% of 469,061 detected panels are successfully matched and retained, while 16.5% are excluded because of missing outputs or panel-key mismatches.The retained dataset contains 391,606 matched panels.
  • Annotation quality: 1.44% of panels contain out-of-taxonomy subtype values, while all 391,606 panels carry a valid visualization category.The invalid subtype values comprise 5,633 panels across 53 unique entries, with Generic Plot accounting for 5,381 panels.
  • Annotation quality: 97.7% of 937 audited matched panels were correctly aligned, with 2.2% ambiguous and 0.1% misaligned, indicating negligible label noise from the letter-based join.The audit verified each sub-caption’s specific details against its assigned crop.
  • Dataset composition: Microscopy dominates the dataset at 27.2%, followed by Generic Plot at 18.7%, Schematic/Diagram at 10.6%, Mechanical Test at 9.2%, and Diffraction at 7.9%.Microscopy is led by SEM at 62.7%, while Simulation is dominated by FEA/FEM Result at 78.4%.

5 Retrieval Experiment

The MatSciFig retrieval experiment trains a dual-encoder with hard-negative mining and evaluates image–text retrieval on a figure-level held-out test split. Fine-tuning substantially outperforms zero-shot CLIP, with performance varying by visualisation category.

  • Model and training: The dual-encoder maps panel images and corresponding summaries or fallback subcaptions into a shared embedding space using CLIP ViT-B/32 and MatSciBERT branches.Both 768-dimensional CLS representations are followed by identical bias-free projection heads.
  • Model and training: Hard-negative mining uses 64 random anchors and 64 mined negatives per 128-panel micro-batch, with gradient accumulation producing an effective batch size of 256.The three-level scheme addresses the limited diversity among panels.
  • Convergence: The average training loss decreases from 2.20 at epoch 0 to 0.11 by epoch 19, while validation RSUM rises from 0.51 to 1.49.RSUM sums R@1, R@5, and R@10 across both retrieval directions, with the fastest gains in the first two epochs.
  • Overall retrieval: Fine-tuning on MatSciFig consistently improves retrieval across all metrics over zero-shot CLIP, whose R@1 is 2.4% i→t and 1.7% t→i.The supplied passage reports the fine-tuned i→t R@1 as 10.5%.
  • Category-level retrieval: Fine-tuning improves performance across all 19 categories, with Table panels achieving the strongest reported result at 60.0% R@1 i→t and an MRR of 0.718.Optical/Photonic and Magnetic/Electronic also achieve R@1 values of 26.8% and 26.1%, respectively.
  • Category-level retrieval: Microscopy is the most challenging category, with 10,624 test panels and the lowest i→t R@1 of 9.2%, while Generic Plot reaches 15.6%.Microscopy is difficult because micrographs are visually homogeneous; Diffraction and Schematic/Diagram also present challenges due to variability.

6 Limitations and Future Work

MatSciFig is limited by restricted article coverage, text-only annotation without direct visual grounding, and a taxonomy that may not capture all materials sub-disciplines. Future work should broaden sources and taxonomy coverage while advancing retrieval beyond the current dual-encoder baseline.

  • Dataset coverage: MatSciFig covers only open-access articles from Elsevier and Springer, potentially biasing coverage toward particular domains and research groups.Adding articles from more publishers would widen the dataset’s coverage.
  • Annotation grounding: Text-only annotation lacks direct visual grounding and depends on the completeness and clarity of figure captions and reference sentences.Ambiguous or insufficiently descriptive captions may produce lower-fidelity annotations, while lower-cost models show hallucination sensitivity.
  • Taxonomy scope: The taxonomy may not fully capture visual diversity beyond alloy, composite, and ceramic literature across materials sub-disciplines.Future work should extend and refine it with domain experts from a wider range of sub-fields.
  • Retrieval model: The CLIP ViT-B/32 and MatSciBERT dual encoder is a straightforward starting point rather than a state-of-the-art system.Future directions include larger vision-language models, cross-attention fusion, and retrieval-augmented generation using MatSciFig’s structured annotations.

7 Conclusion

MatMMExtract is an open-source pipeline that converts compound figures from materials science literature into structured, panel-level multimodal data. Its MatSciFig dataset, selected annotation model, and released resources support AI systems for understanding visual scientific content.

  • Conclusion: MatMMExtract combines structured XML parsing, domain-adaptive compound-figure detection, and taxonomy-guided LLM annotation to generate grounded panel-level image-text pairs at scale.The pipeline uses a manually annotated materials science dataset, MaterialScope, and a two-level visualisation taxonomy.
  • Conclusion: 391,606 annotated panels were extracted from 180,571 source figures, each paired with a grounded sub-caption, visualisation category, sub-category, and concise summary.MatSciFig is described as the largest panel-level multimodal dataset in materials science to date.
  • Conclusion: 4.8% hallucination rate was achieved by Gemini 3.1 Flash Lite, which benchmarking across six large language models identified as the best balance of annotation quality and cost.Gemini 3.1 Flash Lite was selected for downstream use.
  • Conclusion: The MatMMExtract pipeline, MaterialScope annotation dataset, MatSciFig dataset, and retrieval baseline are released as open-source resources.The release is intended to support AI systems capable of understanding and reasoning over materials science literature’s visual content.

A Visualisation Taxonomy

The pipeline classifies materials science sub-panel images using a two-level visualisation taxonomy spanning experimental, computational, and illustrative representations. Domain-specific subtypes cover defined cases, while an “other” category handles panels without an applicable subtype.

  • The taxonomy uses two levels to classify sub-panel images across experimental, computational, and illustrative materials science representations.Each broad category is paired with domain-specific subtypes covering the range of representations encountered in the literature.
  • Domain-specific subtypes are associated with each broad visualisation category to support structured panel classification.
  • The “other” category serves as a fallback when no defined subtype applies.

B Prompt Template

The prompt frames the model as a materials-science scientific figure-analysis expert and instructs it to generate detailed panel-level sub-captions using only captions and reference sentences. A runtime-injected taxonomy block supports structured annotation generation with two-level categories and subtypes.

  • Expertise: The prompt assigns the model expertise in scientific figure analysis specializing in materials science.
  • Panel annotation: It instructs the model to generate detailed sub-captions for each figure panel using only the figure caption and reference sentences from the paper.
  • Taxonomy integration: The complete template injects the full two-level category and subtype taxonomy at runtime for structured annotation generation.
Loading 2606.29667v1…