PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlocking the Visual Record of Materials Science: A Large-Scale Multimodal Dataset from Scientific Literature

Subham Ghosh, Shubham Tiwari, Mohammad Ibrahim, Abhishek Tewari

arXiv:2606.29667v1cs.CVcond-mat.mtrl-scics.AI

TL;DR

Materials-science figures preserve experimental knowledge, but compound figures and caption-only supervision leave panel-level grounding and consistent visual categorization limited. MatMMExtract decomposes compound figures and generates taxonomy-guided annotations, producing MatSciFig with 391,606 annotated panels from 180,571 figures.

  • Problem

    Caption-only supervision lacks panel-level grounding in scientific context, a consistent taxonomy of visualisation types, and summaries of scientific significance.

    Source: §1 · p. 3

  • Method

    MatMMExtract parses open-access articles, localizes and decomposes compound figures, then uses in-text context and a two-level taxonomy to generate panel-level captions, categories, and summaries.

    Source: §2.1 · p. 5, §2.3.1 · p. 8, §7 · p. 23

  • Results

    391,606 annotated panels from 180,571 figures make MatSciFig the largest panel-level multimodal dataset in materials science to date; Gemini 3.1 Flash Lite was selected for annotation, with a 4.8% hallucination rate.

    Source: §2.4.3 · pp. 12–13, §7 · p. 23

  • Takeaways & Limitations

    MatSciFig pairs panels with captions, categories, and summaries, covering figure-based materials knowledge that structured property databases often omit.

    Source: §1 · p. 2, §7 · p. 23

    MatSciFig covers only open-access Elsevier and Springer articles, potentially biasing coverage toward particular domains and research groups.

    Source: §6 · p. 22

Abstract

from arXiv · show

The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.

1 Introduction

Materials science figures preserve experimental context and diverse visual modalities that structured databases and existing vision-language resources do not adequately capture. MatMMExtract decomposes compound figures and enriches panel-level image-text pairs, producing the MatSciFig dataset and supporting resources for materials science vision-language research.

1 Introduction

  • Materials science lacks a large-scale, modality-diverse training corpus, while compound figures and caption-only supervision impede reliable panel-level alignment and scientific grounding.Existing resources are limited in scope, and panel-level summaries and a consistent visualisation taxonomy are also lacking.Source: §1 · pp. 2–3
  • MatMMExtract parses publisher XML into image-caption-reference triplets and decomposes compound figures into panels with grounded sub-captions, category and subtype labels, and summaries.The open-source, license-aware pipeline is released as the matmmextract PyPI package.Source: §1 · p. 3
  • MaterialScope contains 2,811 manually annotated figures and 12,906 boxes, and benchmarks six detectors to show that domain-specific training drives accuracy.Source: §1 · p. 3
  • The work introduces a two-level taxonomy of 19 visualisation categories and benchmarks six low-cost LLMs on classification, caption and summary quality, and hallucination.Source: §1 · p. 3
  • 391,606 annotated panels from 180,571 figures make MatSciFig a large-scale materials science vision-language dataset.The dataset is among the largest figure-level vision-language datasets in materials science.Source: §1 · p. 3
  • A CLIP-style dual encoder combining CLIP ViT-B/32 and MatSciBERT, trained on MatSciFig and indexed with FAISS, provides a reference point for scientific-image research.Source: §1 · p. 3

2 Methods · 2.1 Source Data Collection

MatMMExtract collects open-access materials science articles, decomposes compound figures into sub-panels, and uses taxonomy-guided LLMs to create structured multimodal annotations. The source data are filtered by license and paired with captions and in-text references to support panel-level annotation.

2 Methods

  • MatMMExtract retrieves Elsevier and Springer articles, localises figure sub-panels, and uses an LLM conditioned on captions, reference sentences, and a materials science taxonomy to create the dataset.The final multimodal dataset combines sub-panel detection with structured annotation generation.Source: §2 · p. 4, Figure 1 · p. 4

2.1 Source Data Collection

  • The corpus draws metadata from OpenAlex [27] and Scopus, but is restricted to CC-BY and CC-BY-NC articles to support downstream redistribution.OpenAlex permits filtering by specific Creative Commons license, whereas Scopus labels open-access articles as gold or hybrid, which may include licenses restricting derivative works.Source: §2.1 · pp. 4–5
  • For Elsevier and Springer journals, structured XML yields figure links, captions, and in-text reference sentences; graphical abstracts without captions are removed.The remaining entries form image-caption-reference triplets, providing richer context than caption-only pairs.Source: §2.1 · p. 5
  • 62% of figures in the curated corpus are compound, making direct image-text alignment difficult because one caption describes multiple sub-images.The pipeline addresses this by treating figures as single images or compound figures before annotating individual sub-panels.Source: §2.1 · p. 5
  • Processing first classifies figures as single images or compound figures, then generates structured annotations for each sub-panel using the figure caption and in-text reference sentence.Compound figures contain multiple sub-panels, often labelled (a), (b), (c), and (d).Source: §2.1 · p. 5
  • Each atomic image receives a caption grounded in the relevant original caption and reference sentence, a visualisation category and sub-category, and a summary of its content and scientific significance.The categories distinguish image types such as microscopy (SEM or TEM) and diffraction (XRD or EBSD Map).Source: §2.1 · p. 5

2.2 Compound Figure Detection

The section addresses compound-figure panel detection by benchmarking detectors, curating a materials-science-specific dataset, and evaluating domain-adapted models. YOLO12-m is selected for the annotation pipeline based on its accuracy and balance of model size and speed.

2.2 Compound Figure Detection

  • On FigEx annotations, six detectors achieved APf of 0.67–0.72, with lower performance on materials-science figures, indicating a domain gap.The initial training set contained 6,454 images, with 717 validation images; detectors were trained for 50 epochs.Source: §2.2 · p. 5

2.2.1 Curated Dataset

  • MaterialScope contains 2,811 manually annotated figures and 12,906 bounding boxes; shared-image annotation agreement was κ = 0.9245, and the dataset is split into 2,249 training, 281 validation, and 281 test figures.Two domain-expert PhD researchers annotated figures from materials domains including alloys, composites, ceramics, steels, polymers, and thin films.Source: §2.2.1 · p. 6

2.2.2 Domain-Adaptive Training

  • The six architectures were retrained on MaterialScope for 50 epochs; YOLO models used 1024 × 1024 resolution and batch size 16, while DAB-DETR used batch size 8 with gradient accumulation of 2.YOLO models used pretrained weights, SGD, and cosine learning-rate decay; DAB-DETR used AdamW, warmup, cosine decay, and early stopping.Source: §2.2.2 · p. 6

2.2.3 Experimental Setup

  • Evaluation used the held-out test split at confidence 0.55, NMS IoU 0.45, and 1024 × 1024 resolution, with precision, recall, and F1 computed using greedy matching at IoU 0.50.The protocol also reports AP@50:95 for frequent classes A–E and common classes F–K plus single-panel figures, and measures throughput using single-image batches.Source: §2.2.3 · p. 7

2.2.4 Detection Results

  • YOLO12-m achieves the best retrained-model mAP50 (0.9227 versus 0.9028 for YOLO8-m) and is adopted for its balance of accuracy, size, and speed.It has 20.15 M parameters and 86.82 GFLOPs.Source: Table 1 · p. 7, §2.2.4 · pp. 7–8
  • DAB-DETR has the lowest mAP50 (0.6569) and slowest throughput (9.18 img/s), while YOLO8-m is fastest (14.31 img/s) and YOLO10-m has highest precision (0.9701).YOLO10-m recall is 0.9023; the authors favor recall for compound-figure parsing because missed panels are more costly than spurious boxes.Source: §2.2.4 · p. 8

2.3 Caption, Category, and Summary Generation

MatMMExtract generates text-grounded panel annotations from the original figure caption and in-text reference sentence, combining a sub-caption, taxonomy category and subtype, and scientific summary. A constrained prompting and output process structures these annotations without passing the panel image to the LLM.

2.3 Caption, Category, and Summary Generation

  • For each panel, the LLM produces a sub-caption, visualisation category and subtype, and scientific summary, which form a structured record paired with the cropped image.The model receives only the figure caption and in-text reference sentence; the panel image itself is never passed to it.Source: §2.3 · p. 8, §2.3.2 · p. 9

2.3.1 Taxonomy Design

  • The two-level taxonomy defines 19 broad visualisation categories with domain-specific subtypes spanning experimental, computational, and illustrative materials science figures.Category and subtype are strict enumerations, with “other” reserved as a fallback when no defined subtype applies.Source: §2.3.1 · p. 8

2.3.2 Annotation Generation

  • The prompt supplies the full taxonomy and grounds sub-captions in the caption and reference sentence, while a JSON schema enforces required, well-formed fields.Group-level references apply across panels, while panel-specific references are scoped to their sub-panel; summaries are limited to 40–60 words and drawn only from that panel’s sub-caption and tied references. Generation uses temperature 0.1 and disables chain-of-thought reasoning.Source: §2.3.2 · pp. 8–9

2.4 Benchmarking

Across six language models, Gemini models led on annotation quality, while Gemini 3.1 Flash Lite was selected for downstream generation because it combined strong quality with the lowest observed hallucination rate and modest cost.

2.4 Benchmarking

  • A custom Flask-based annotation tool benchmarked six large language models for visualisation category and sub-category classification and generated sub-caption and summary quality, with outputs reviewed alongside compound figures and source text.Generation quality was benchmarked on 40 compound figures, yielding 1,311 panel-level sub-caption/summary pairs across the six models; the outputs were generated from captions and in-text reference sentences. A single materials science PhD student assessed each output, defining hallucination as information absent from or not inferable from those sources.Source: §2.4 · p. 9, §2.4.3 · pp. 11–12

2.4.1 Category Classification

  • Gemini 3.5 Flash led category classification, scoring 99.0% on Microscopy, 97.8% on Mechanical Test, and 95.5% on Thermal Analysis.Gemini 3.1 Flash Lite was close behind, with 94.4% on Simulation and 92.6% on Schematic/Diagram; Photograph was hardest, and GPT-5.4 Nano scored 43.2% there.Source: §2.4.1 · pp. 9–10, Figure 3 · p. 10

2.4.2 Sub-Category Classification

  • Gemini 3.5 Flash and DeepSeek V4 Flash were the most reliable for subcategory classification, while GPT-5.4 Nano was weakest on fine-grained distinctions.XPS Spectrum and FTIR Spectrum were perfect or near-perfect across models, and all models exceeded 98% on SEM; on Fatigue/S-N Curve, Gemini 3.5 Flash scored 100% versus 11.5% for GPT-5.4 Nano. Results for subcategories with n < 30 are indicative only because of wide uncertainty.Source: §2.4.2 · pp. 10–11, Figure 4 · p. 11

2.4.3 Sub-Caption and Summary Generation

  • Gemini 3.1 Flash Lite achieved 82.0% good sub-captions, 83.5% good summaries, and a 4.8% hallucination rate—the lowest observed and the only model’s full interval below 10%.Its caption and summary quality intervals overlap Gemini 3.5 Flash’s, making the two statistically indistinguishable on generation quality; Lite was chosen for downstream annotation because it cost a fraction as much to run.Source: Figure 5 · p. 12, §2.4.3 · pp. 12–13

2.4.4 Inference Infrastructure and Cost

  • Gemini 3.1 Flash Lite was selected for downstream annotation at $0.25 per million input tokens, balancing generation quality and hallucination rate against cost.DeepSeek V4 Flash and GPT-5.4 Nano were the lowest-cost options at $0.19 and $0.20 per million input tokens, respectively; Gemini 3.5 Flash cost $1.50 per million input tokens and $9.00 per million output tokens.Source: Table 2 · p. 13, §2.4.4 · p. 13

3 The MatmmExtract Package

MatMMExtract implements the complete pipeline as an open-source Python package, organized around its six stages.

3 The MatmmExtract Package

  • The complete pipeline is implemented in the open-source matmmextract Python package, available on PyPI and installable with pip install matmmextract.The package requires Python ≥3.10.Source: §3 · p. 14
  • The package is organized into six modules corresponding to the pipeline stages, with documentation and usage examples in the code repository.Source: §3 · p. 14

4 Dataset

MatSciFig converts compound materials-science figures into panel-level annotations at scale, with post-processing and dataset statistics describing its quality, content, and annotation lengths.

4 Dataset

  • Unlike EXSCLAIM!, which is panel-level but microscopy-only and keyword-based, MatSciFig spans visualisation modalities with taxonomy labels and grounded natural-language descriptions.Prior materials-science datasets include benchmarks for evaluation [22] [23] and knowledge extraction, but do not provide this large-scale panel-level resource for training vision-language models.Source: §4 · p. 14

4.1 MatSciFig

  • MatSciFig contains 391,606 panels from 180,571 figures, each paired with a sub-caption, visualisation category and sub-category, and extended summary.It is an order of magnitude larger than prior materials-science figure datasets.Source: §4.1 · pp. 14–15, §4 · p. 14

4.2 Data Quality and Post-Processing

  • Of 469,061 detected panels, 391,606 (83.5%) were retained; 16.5% were excluded, mostly because of panel-key mismatches.The two exclusion reasons were missing LLM output files (2.7%) and panel-key mismatches (13.8%); mismatches often involved panels without caption labels.Source: §4.2 · p. 15
  • All 391,606 panels have a valid visualisation category, and 97.7% of 937 audited matches were correctly aligned.Out-of-taxonomy subtypes affected 1.44% of panels (5,633) and were retained in training; 2.2% of audited matches were ambiguous and 0.1% misaligned.Source: §4.2 · p. 15

4.3 Dataset Statistics

  • Microscopy is the largest visualisation category at 27.2%, and SEM accounts for 62.7% of microscopy panels.Other prominent categories are Generic Plot (18.7%) and Schematic/Diagram (10.6%); Line Graph makes up 47.6% of Generic Plot, EBSD Map 41.7% of Diffraction, and FEA/FEM Result 78.4% of Simulation.Source: Figure 6 · p. 16, §4.3 · p. 16, Figure 7 · p. 17
  • Sub-captions centre around 15–20 words, while summaries concentrate between 45 and 55 words.Sub-captions follow source-caption length variation; summaries are more uniform because generation imposes a 40–60 word limit.Source: §4.3 · p. 17, Figure 8 · p. 18

5 Retrieval Experiment

MatSciFig supports a dual-encoder retrieval model trained on panel–description pairs, and fine-tuning it yields substantial gains over zero-shot CLIP. Performance varies across visualisation categories, reflecting differences in how visually and textually distinctive they are.

5.1 Model Architecture

  • The image encoder is CLIP ViT-B/32 and the text encoder is MatSciBERT; both project 768-dimensional CLS features into a shared 512-dimensional embedding space.Panel summaries are used as text, with subcaptions as fallback when summaries are unavailable.Source: §5.1 · p. 18

5.2 Training Objective

  • Symmetric InfoNCE trains both image-to-text and text-to-image matching, and the learnable temperature converges to 0.017 from an initial 0.07.True pairs are diagonal entries in the similarity matrix, while off-diagonal entries are in-batch negatives.Source: §5.2 · pp. 18–19

5.3 Hard Negative Sampling

  • Hard negatives are prioritized from the same subtype but a different figure, then the same category, then random panels, so training emphasizes confusable examples.Each micro-batch contains 128 panels, and gradient accumulation of 2 gives an effective batch size of 256; most negatives come from the first two levels.Source: §5.3 · p. 19

5.4.3 Convergence

  • Validation RSUM rises from 0.51 at initialisation to 1.49 at epoch 19, while average training loss falls from 2.20 at epoch 0 to 0.11.The most rapid validation gains occur in the first two epochs of the 20-epoch training run.Source: §5.4.3 · p. 20

5.4.1 Data Split

  • Evaluation uses a held-out test split of 39,315 panels in both image-to-text and text-to-image directions, reporting R@K (K ∈ {1, 5, 10, 50, 100}), MRR at depth 1000, and mAP@100.Panel embeddings are indexed using FAISS for exact innerproduct search on L2-normalised embeddings, equivalent to cosine similarity retrieval.Source: §5.5 · p. 20

5.6.1 Overall Retrieval Performance

  • 4.4× and 5.4×: fine-tuning on MatSciFig raises R@1 over zero-shot CLIP to 10.5% (i→t) and 9.2% (t→i), respectively.R@10 reaches 38.6% and 36.7%, while R@100 reaches 69.7% and 68.7%; i→t MRR rises from 0.043 to 0.196.Source: Table 6 · p. 21, §5.6.1 · p. 21

5.6.2 Per Category Retrieval Performance

  • Fine-tuning consistently improves retrieval over zero-shot CLIP across all 19 categories in both directions; Table panels lead at 60.0% i→t R@1, while Microscopy is lowest at 9.2%.Optical/Photonic and Magnetic/Electronic reach R@1 values of 26.8% and 26.1%, respectively; Microscopy is the largest test category (n = 10,624).Source: §5.6.2 · p. 21
  • Diffraction and Schematic/Diagram are also challenging, with R@1 of 11.7% and 13.8%, while Generic Plot reaches 15.6% despite 7,305 test panels.The passages attribute difficulty to large intra-category visual variability and suggest that distinctive structural cues in line graphs and bar charts help Generic Plot retrieval.Source: §5.6.2 · p. 22

6 Limitations and Future Work

MatSciFig is limited by its source coverage and text-only annotation process; future work should broaden the taxonomy and explore stronger retrieval architectures.

Limitations and Future Work

  • MatSciFig covers only open-access Elsevier and Springer articles, potentially biasing coverage toward particular domains and research groups; adding publishers could widen it.Source: §6 · p. 22
  • Annotations are generated from text without visual grounding, so their fidelity depends on caption and reference-sentence clarity, with ambiguous captions and lower-cost models especially vulnerable to errors.Benchmarking shows high-quality annotations for most panels, but text-only generation remains sensitive to the completeness and clarity of the source descriptions.Source: §6 · p. 22
  • The taxonomy targets common visualisation types in alloy, composite, and ceramic literature, may miss other materials sub-disciplines, and could be refined with broader expert input.Source: §6 · p. 22
  • The CLIP ViT-B/32–MatSciBERT dual encoder is a starting point, motivating exploration of larger vision-language models, cross-attention fusion, and retrieval-augmented generation.The proposed retrieval-augmented approaches could leverage MatSciFig’s structured annotations.Source: §6 · pp. 22–23

7 Conclusion

MatMMExtract turns materials science figures into structured panel-level image-text pairs at scale, producing the MatSciFig dataset and releasing the pipeline, datasets, and retrieval baseline as open resources.

7 Conclusion

  • MatMMExtract combines publisher XML parsing, domain-adaptive compound-figure detection trained on MaterialScope, and LLM annotation guided by a two-level visualisation taxonomy.The pipeline transforms raw scientific figures into richly annotated image-text pairs at scale.Source: §7 · p. 23
  • Gemini 3.1 Flash Lite achieved a 4.8% hallucination rate, while MaterialScope-fine-tuned YOLO12-m achieved 0.9227 mAP50; retrieval fine-tuning improved over zero-shot CLIP across all 19 visualisation categories.Gemini 3.1 Flash Lite was selected for downstream annotation tasks; retrieval results support MatSciFig’s utility for scientific vision-language research.Source: §7 · p. 23
  • MatSciFig contains 391,606 annotated panels from 180,571 figures, each paired with a grounded sub-caption, visualisation category and sub-category, and concise summary.The conclusion describes it as the largest panel-level multimodal dataset in materials science to date.Source: §7 · p. 23
  • The authors release the MatMMExtract pipeline, MaterialScope annotation dataset, MatSciFig dataset, and retrieval baseline as open-source resources.These resources are intended to support AI systems for understanding and reasoning over materials science literature’s visual content.Source: §7 · p. 23

A Visualisation Taxonomy

The annotation pipeline classifies sub-panel images with a two-level visualisation taxonomy whose domain-specific subtypes cover experimental, computational, and illustrative representations in materials science literature.

A Visualisation Taxonomy

  • The pipeline uses a two-level visualisation taxonomy to classify sub-panel images.Each broad category is associated with domain-specific subtypes.Source: A Visualisation Taxonomy · p. 28
  • The taxonomy’s subtypes cover experimental, computational, and illustrative representations encountered in materials science literature.Source: A Visualisation Taxonomy · p. 28
  • “Other” serves as a fallback when no defined subtype applies.Source: Table 8 · p. 29

B Prompt Template

The prompt template supports structured annotation generation, with the full two-level category and subtype taxonomy injected at runtime.

B Prompt Template

  • The structured annotation prompt uses a runtime-injected taxonomy listing its full two-level categories and subtypes.Figure 9 presents the complete prompt template.Source: Figure 9 · p. 30

Related papers

Loading 2606.29667v1…