PaperlayerView exact revision on arXiv

Source-linked AI summary

From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding

Raul Ortega, José Manuel Gómez-Pérez

arXiv:2609.00948v1cs.CVcs.AIcs.CL

TL;DR

Scientific diagram understanding remains challenging because diagrams encode abstract relationships and processes, yet VLM training data sparsely covers them. The paper introduces terminology-grounded generation of diagrams, captions, and questions, and reports broad benchmark gains with SciGram, including improved existing models.

  • Problem

    Scientific diagram understanding remains an open problem because diagrams require contextual reasoning over symbolic, abstract, and relational content.

  • Method

    The framework extracts curriculum terminology, generates atomic facts, retrieves web diagrams, and synthesizes captions and multiple-choice instructions.

  • Results

    SciGram models substantially improve TQA, SQA, and AI2D performance, outperforming or matching state-of-the-art VLMs with fewer training instances.

  • Takeaways & Limitations

    Terminology-grounded instruction generation provides a practical strategy for developing scientific diagram-understanding models.

  • Takeaways & Limitations

    SciGram retains limitations from noisy web data and synthetic supervision, including prior-knowledge-solvable MCQs and labeling inconsistencies.

Abstract

from arXiv · show

Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images. However, they continue to struggle with scientific diagrams, which are designed to convey functional or relational meaning rather than literal scenes. We therefore introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula. Our approach systematically extracts domain concepts, synthesizes atomic facts, retrieves relevant diagrams from the web, and generates multimodal supervision in the form of diagram captions and multiple-choice questions. Using this pipeline, we construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences. Despite relying on noisy web data and synthetic annotations, models fine-tuned on SciGram achieve substantial improvements on diagram-centric benchmarks, including TQA, ScienceQA, and AI2D, outperforming or matching state-of-the-art VLMs while using fewer training instances. Furthermore, augmenting existing models such as LLaVA OneVision with SciGram establishes new state-of-the-art performance on diagram question answering. Our results highlight the effectiveness of terminology-grounded instruction generation as a general strategy for improving vision-language reasoning in scientific domains. To support future research in scientific diagram understanding, we release both the SciGram dataset and models.

1 Introduction

Scientific diagram understanding remains difficult because diagrams encode abstract relationships and processes, while existing VLM training data contains few such examples. The paper addresses this gap with terminology-driven SciGram data and LLaVA-based models.

  • Motivation: Scientific diagram understanding remains largely unsolved despite multimodal benchmarks such as AI2D, TQA, and ScienceQA.These benchmarks require reasoning over scientific diagrams alongside textual questions.
  • Motivation: Scientific diagrams are symbolic, abstract, and structurally diverse, conveying concepts, relationships, or processes rather than literal scenes.Their interpretation requires grounding in scientific context.
  • Approach: The terminology-driven framework extracts curriculum concepts, generates atomic facts, retrieves web diagrams, and synthesizes diagram-grounded instructions.The resulting pipeline connects scientific terminology with visual supervision for VLM fine-tuning.
  • Resources: 194,071 scientific diagrams comprise SciGram, paired with captions and multiple-choice questions in instruction-following format.The dataset prioritizes coverage over precision and includes over 1.4M synthetic instructions.
  • Models and evaluation: SciGram models are fine-tuned LLaVA-based VLMs evaluated across scientific diagram-understanding benchmarks.The evaluation reports that these models outperform or match state-of-the-art VLMs and frontier models.

2 Related work

Scientific diagram understanding has evolved from diagram-specific parsers and reasoning models toward multimodal VLMs, but general-purpose instruction data still sparsely covers scientific diagrams.

  • Early approaches: Early approaches applied machine reading comprehension, visual question answering, and diagram-specific parsers to scientific diagram understanding.These approaches highlighted challenges distinct from those posed by natural images.
  • Structured reasoning: Later reasoning-focused and graph-based models targeted spatial and semantic relations in diagrams.These methods extended the field beyond initial machine reading and visual question answering approaches.
  • Multimodal models: Transformer-based multimodal models such as VL-BERT and LXMERT primarily focused on natural images, with ISAAQ partially addressing diagram question answering.This left scientific diagram coverage limited in earlier multimodal work.
  • Modern VLMs: Modern VLMs use image–text alignment and large language models but rely on general-purpose instruction datasets with sparse scientific-diagram coverage.This motivates domain-specific instruction data for scientific diagram understanding.

3 Method

The method builds diagram-grounded instruction data through a structured progression from curriculum terminology to atomic facts, retrieved diagrams, and multimodal instructions.

  • Pipeline: The framework uses four stages: terminology extraction, atomic fact generation, diagram retrieval, and instruction generation.These stages produce complementary datasets aligned with different VLM training needs.
  • Terminology extraction: Scientific terminology is extracted from middle-school natural-science textbooks to provide the semantic backbone for downstream data generation.The source is organized by topics, lessons, explanations, and instructional materials.
  • Terminology extraction: Noun phrases are filtered using a weirdness-index threshold of t = 2, lemmatized, embedded with RoBERTa-base, and clustered by Euclidean distance.Terms beyond one standard deviation from their topic centroid are discarded.
  • Terminology extraction: 4,820 distinct scientific terms remain after terminology curation.The resulting vocabulary is intended to be semantically coherent and domain-relevant.
  • Atomic fact generation: Atomic facts provide fine-grained semantic anchors linking curriculum concepts to relevant diagrams.For each topic, LLaMA3-8B-Instruct generates concise middle-school-level statements over terminology combinations, producing 5,508,218 unique facts after deduplication.
  • Diagram retrieval: Web images are retrieved by appending “diagram” to each atomic fact and lightly filtered using fact linkage, perceptual hashing, and file validation.The filtering preserves scale while accepting residual noise from natural images, irrelevant visuals, and stylistic artifacts.
  • Instruction generation: Qwen2-VL-7B generates captions and visually grounded MCQs, while existing diagram and text QA datasets form the SciGram-M3 subset.Captions emphasize components and relationships; MCQs are middle-school level, visually grounded, deduplicated, and balanced across answer options.

4 The SciGram dataset

SciGram combines compact, staged training subsets with large-scale synthetic supervision, achieving generally strong annotation quality while retaining measurable web-data noise and question limitations.

  • Dataset composition: SciGram-Align contains 582,213 captioning pairs, with three detailed captions per diagram for visual–text alignment.The captions describe entities and processes in each image.
  • Dataset composition: SciGram-VIT contains 737,887 diagram MCQA instructions, each with four options and one correct answer.This subset targets instruction tuning for diagram-based multiple-choice question answering.
  • Dataset composition: SciGram-M3 contributes 47,506 instructions from TQA, SQA, OpenBookQA, ARC-Easy/Challenge, and the LLaVA OV AI2D selection.The subset provides additional domain-focused training data aligned with target tasks.
  • Quality evaluation: Four experts reviewed 600 items, yielding 82.41% raw agreement, Cohen’s κ of 0.27, and average Gwet’s AC1 of 0.59.The authors use Gwet’s AC1 because class-prevalence imbalance makes κ less suitable here.
  • Quality evaluation: 24% of retrieved images are not actual diagrams, while 88% of captions align with diagrams and 89% of MCQs are visually grounded.Other reported quality rates include 82% key-element coverage, 93% unambiguous MCQs, and 92% distinctive distractors.
  • Quality limitations: 61% of MCQs may be answerable from prior knowledge and 16% contain labeling inconsistencies.The paper calls for stronger diagram filtering and automated consistency verification.

5 Experimental Setup

The evaluation uses LLaVA-based models trained from scratch or fine-tuned from pretrained LLaVA-OV, testing them on complementary scientific diagram benchmarks.

  • Models and training: Two training regimes produce LLaVA-SciGram 7B models: SciGram training from scratch and fine-tuning of pretrained LLaVA-OV 7B.Each model requires approximately 450 GPU-hours on two NVIDIA A100 GPUs.
  • Models and training: LLaVA-SciGram 7B uses a pretrained CLIP encoder and Qwen2-Instruct 7B language backbone, with alignment followed by LoRA instruction tuning.The alignment stage trains only the projection matrix while freezing the vision encoder and language model.
  • Models and training: LLaVA-SciGram OV 7B fine-tunes pretrained LLaVA OV weights using a SIGLIP encoder and Qwen2-Instruct 7B.
  • Benchmarks: Evaluation covers TQA, SQA, and AI2D, selected to represent varied grade levels, modalities, and reasoning types.Benchmark test diagrams are excluded from SciGram to prevent contamination from web-sourced images.
  • Benchmarks: TQA combines text-only and diagram-grounded questions across physical, life, and earth sciences.Questions without diagrams use the associated lesson as context.
  • Benchmarks: SQA includes diagram, text-only, and natural-image questions from elementary and high school science curricula.
  • Benchmarks: AI2D tests grade-school science diagrams with masked labels and also includes a visible-label variant.The masked-label split requires models to infer elements and processes visually.

6 Results

SciGram fine-tuning improves LLaVA performance across scientific diagram benchmarks, with the strongest gains on visually supported questions and diagram-reasoning categories.

  • 6.1 Effect of SciGram in LLaVA models: SciGram fine-tuning consistently boosts performance, with gains up to 16 points across evaluated benchmarks.LLaVA-SciGram OV generally achieves the largest improvements, especially on visually supported questions.
  • 6.2 Comparison with other models: LLaVA-SciGram OV establishes a new state of the art on TQA diagram multiple-choice questions, outperforming all baselines.GPT4o has the highest overall TQA accuracy, driven largely by text-only questions.
  • 6.2 Comparison with other models: 0.54% and 2.92% are the SQA IMG gains for LLaVA-SciGram and LLaVA-SciGram OV over the previous best, respectively.LLaVA-SciGram OV reaches accuracy comparable to T-SciQ, a model specialized for SQA with chain-of-thought reasoning.
  • 6.2 Comparison with other models: 1.05% is LLaVA-SciGram OV’s margin over MOLMo on AI2D’s opaque-label split.On the transparent-label split, GPT4o leads, while LLaVA-SciGram OV performs strongest overall across both splits.
  • 6.3 Ablation study: SciGram data improves SQA and AI2D consistently across fine-tuning stages, while TQA DQ remains above baseline early and on par finally.The final stage uses substantially fewer instructions than the baseline.
  • 6.3 Ablation study: The full Align, VIT, and M3 pipeline gives the strongest subset-combination results, indicating contributions from all subsets.Adding OpenBookQA and ARC to SciGram-M3 produces small but consistent cross-benchmark gains.
  • 6.4 Knowledge and reasoning analysis: LLaVA-SciGram OV gains over five points on Structure, Teleology/Purpose, Algebraic, Spatial/Kinematic, and Visual Labeling question types.SciGram models match or surpass the baseline across most knowledge and reasoning types, except Processes & Causal and Causal/Explanation.
  • 6.5 Visual support analysis: LLaVA-SciGram OV outperforms LLaVA OV by nearly ten points on questions requiring visual reasoning but matches it on language-prior questions.Progressive fine-tuning across SciGram subsets yields the strongest performance with the full pipeline.

7 Conclusions and future work

The paper presents SciGram as a terminology-grounded framework and large-scale dataset for scientific diagram instruction generation. Despite noise and minor inconsistencies, SciGram improves diagram understanding and supports strong multimodal supervision.

  • SciGram contains over 194k diagrams and 1.4M visual instructions generated from curriculum-derived scientific terminology.The dataset spans the natural sciences and includes diagram-grounded supervision.
  • Models fine-tuned on SciGram substantially improve performance on TQA, SQA, and AI2D while using substantially fewer training instances.The reported comparison is against existing state-of-the-art vision-language models.
  • Further training of LLaVA OneVision with SciGram establishes new state-of-the-art performance in diagram-based question answering.
  • SciGram improves visual grounding and diagram-centric reasoning rather than merely textual knowledge, according to the reported results.
  • Future work targets diagram filtering, caption alignment, question generation, and factual accuracy of synthesized scientific claims.The methodology may also extend to other structured visual knowledge sources without costly manual annotation.

Ethics Statement

The ethics statement addresses licensing, bias, environmental impact, and misuse risks associated with SciGram and similar systems. It limits intended use to educational and research purposes and urges responsible deployment.

  • Users are responsible for complying with dataset and pretrained-model licenses because improper use may create legal or ethical violations.
  • SciGram links images owned by their respective copyright holders and is released under CC BY 4.0 for educational and research purposes.The authors state that they do not host or redistribute the linked images.
  • Pretrained-model biases may affect downstream outputs by disadvantaging groups or misrepresenting information.The statement recommends considering these risks when deploying similar models.
  • Training and fine-tuning large models are computationally expensive and contribute to carbon emissions.The authors encourage efficient training strategies and consideration of environmental costs.
  • The approach could be misused for automated content generation or misinformation despite its research and educational intent.The authors recommend safeguards and ethical guidelines to minimize potential harm.

Reproducibility Statement

The paper releases code, data access information, and prompt templates for dataset generation and evaluation. Reproducibility is constrained by URL-based image distribution, resource requirements, proprietary APIs, and web-data examples.

  • The authors release code for SciGram generation, model training, and evaluation, with data and models hosted through linked repositories.
  • SciGram distributes image URLs rather than image files, so licensing restrictions may cause some images to become inaccessible over time.
  • Large pretrained models require substantial fine-tuning and inference resources, limiting accessibility for researchers with limited hardware.
  • Proprietary evaluation APIs may incur costs, change, or disappear, making exact reproduction and long-term comparison challenging.
  • The appendix documents prompts for atomic facts, captions, multiple-choice questions, and diagram-question answering.The prompts specify scientific domains, middle-school difficulty, visual grounding, and structured outputs.
  • Caption generation emphasizes diagram components, relationships, and visible spatial or temporal information.
  • Multiple-choice generation uses four answer choices and requires questions to rely on diagram elements at middle-school level.
  • The appendix provides examples of generated captions and question-answer pairs, including scientific processes and visual identification tasks.

D Terminology stats

SciGram’s terminology set contains 4,820 extracted terms shaped by frequency, domain distinctiveness, and cross-subject overlap. Filtering and balancing procedures address long-tail terminology and answer-choice imbalance.

  • 4,820 extracted terms form the final terminology set, including 1,295 terms appearing only once in the TQA textbook.The weirdness-index filter retains contextually important infrequent terms and eliminates 15.09% of candidate noun phrases.
  • 15.09% of candidate noun phrases were eliminated by the weirdness-index filter.
  • Infinite-weirdness terms are excluded from the highest and lowest score comparison, while Cellular Respiration and Epicenter receive high scores.
  • Physical Sciences are slightly underrepresented, while 399 terms overlap with Earth Sciences and 432 overlap with Life Sciences.
  • Nearly 16% of terms appear in two subjects and 183 are shared across all three, combining subject specificity with transversal curriculum coverage.
  • SciGram-M3 balances correct-answer choices by shuffling options to reduce bias and overfitting during training.

F Human evaluation of SciGram

Human evaluation assessed diagram, caption, and MCQ quality using expert annotation and agreement measures. Captions generally supported diagram interpretation, while MCQ wording and diagram grounding received strong agreement.

  • Evaluation setup: Four NLP experts evaluated 200 diagrams, 200 caption–image pairs, and 200 diagram-based multiple-choice questions each.The assessment covered diagram suitability and interpretability, caption quality, and MCQ grounding, difficulty, wording, distractors, answer choices, and correctness.
  • Caption quality: 0.8553 Gwet AC1 agreement was reported for captions covering diagram elements, relations, and processes.
  • Caption quality: 0.4004 Gwet AC1 agreement was reported for captions helping readers interpret diagrams rather than merely describe them.
  • MCQ quality: 0.8485 Gwet AC1 agreement indicated that questions were grounded in their diagrams.
  • MCQ quality: 0.8658 Gwet AC1 agreement was reported for precise, unambiguous question wording.
  • MCQ quality: 0.787 Gwet AC1 agreement was reported for correctness of the selected answers.

H Evaluation details

The evaluation details describe which benchmark results came from prior literature versus custom prompting and identify supplementary figures and tables used to document the evaluation and pipeline.

  • Benchmark evaluation: Reported benchmark results combined literature values with evaluations performed using custom prompts when official results were unavailable.
  • Benchmark evaluation: MemN+VQA and MemN+DPG on TQA used literature results.
  • Supplementary materials: Table 9 lists the hyperparameters used across SciGram pipeline stages.
  • Benchmark evaluation: Phi-3 Vision, MOLMo 7B-D, and Pixtral 12B used a mixture of custom-prompt and literature results across TQA, ScienceQA, and AI2D settings.
Loading 2609.00948v1…