PaperlayerView exact revision on arXiv

Source-linked AI summary

DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context

Ling Yue, Tingwen Zhang, Jiaying Wang, Zhen Xu, Shaowu Pan

arXiv:2604.20857v2cs.IRcs.AI

TL;DR

Existing scientific-figure resources often mix schematic diagrams with other visual materials and omit the document context needed to interpret them. DiagramBank addresses this gap with a quality-audited, context-rich dataset and reports 93.67% precision for its final release.

  • Problem

    Existing scientific-figure resources often mix schematics with plots, photos, tables, or screenshots and omit body-text context explaining figure use.

  • Method

    DiagramBank combines automated PDF parsing, figure classification, context extraction, and a VLM cascade to release diagrams with multi-level metadata, provenance, labels, and confidence views.

  • Results

    93.67% precision was estimated for the final release, while the dataset contains 57,100 cascade-filtered scientific schematic diagrams.

  • Takeaways & Limitations

    The released records support scientific-document understanding, diagram retrieval, corpus analysis, and benchmark-oriented protocols with auditable provenance and filtering choices.

  • Takeaways & Limitations

    The precision-oriented release may exclude useful lower-confidence diagrams, coverage is limited to accessible OpenReview venues and years, and downstream utilities remain qualitative sanity checks.

Abstract

from arXiv · show

Scientific papers use schematic diagrams to communicate methods, workflows, and system structure, yet existing scientific-figure corpora often mix them with plots, screenshots, and photographs and rarely preserve document context. We introduce DiagramBank, a quality-audited dataset of 57,100 schematic diagrams curated from OpenReview-hosted AI/ML venues. Each record links a diagram image to its paper title, abstract, figure caption, in-text figure-reference spans, venue/year metadata, provenance fields, and filtering labels. DiagramBank is a reusable resource for scientific-document understanding, diagram retrieval, corpus analysis, and future benchmark construction. We describe its extraction and cascade-filtering pipeline, release schema, confidence-controlled views, dataset card, and indexing utilities. A manual blind audit of the released cascade-filtered records estimates 93.67% precision, and a separate CLIP threshold analysis characterizes the precision--coverage trade-off for simpler filtering views. We further provide lightweight metadata-indexing and authoring examples to illustrate downstream protocols without treating these utilities as standalone methods. The code is public at: https://github.com/csml-rpi/DiagramBank.

1 Introduction

DiagramBank addresses the lack of diagram-specific scientific-figure resources that preserve document context and filtering provenance. It releases 57,100 quality-audited schematic diagrams with linked paper, figure, and attribution metadata.

  • Motivation: Schematic diagrams encode objects, modules, information flow, layout, arrows, labels, grouping, and community-specific notation.Their structure motivates treating diagram understanding as distinct from ordinary plot or natural-image understanding.
  • Research gap: Existing scientific-figure resources often mix schematics with plots, photos, tables, or screenshots and omit explanatory body-text passages.DiagramBank is designed to preserve paper-level and figure-level context, filtering uncertainty, and provenance for audit and attribution.
  • Contribution: DiagramBank contains 57,100 cascade-filtered scientific schematic diagrams from OpenReview-hosted AI/ML venues.Each record couples an image with title, abstract, caption, in-text figure-reference spans, venue/year metadata, quality-control labels, and attribution fields.
  • Contribution: The release provides a diagram-specific corpus, measured filtering quality, confidence views, provenance, a dataset card, indexing code, and reporting guidance.The high-confidence CLIP view reaches 94.4% precision at τ = 0.85, while the final release reaches 93.67% precision.

2 Related Work and Resource Positioning

DiagramBank is positioned as a diagram-specific, auditable resource that fills gaps left by heterogeneous figure corpora and supports context-aware retrieval and evaluation. It supplies multi-level evidence rather than introducing a new downstream algorithm.

  • Existing resources: Prior resources support captioning, figure classification, summarization, or visual-summary selection but often mix diagrams with other figure types.Examples include SciCap, DocFigure, ACL-Fig, and representative-figure ranking work.
  • Emerging need: Automated authoring and diagram-generation systems increase demand for scientific-figure resources that can support grounded workflows.The cited systems include AI Scientist, autonomous paper generation, Paper-Banana, AutoFigure, DiagrammerGPT, and Sci-Doc2Diagrammer.
  • Resource positioning: DiagramBank preserves diagram retrieval units, multi-granularity textual evidence, paper metadata, and measured filtering quality for auditable evaluations.It is explicitly positioned as neither a benchmark leaderboard nor a new retrieval algorithm.

3 Dataset Construction and Quality Assurance

DiagramBank constructs its resource through normalized metadata and figure-context extraction, followed by CLIP routing and a VLM cascade for schematic classification. Complementary blind audits quantify both confidence-view and final-release precision.

  • Pipeline: The four-stage pipeline comprises metadata acquisition, content extraction, schematic classification, and relational aggregation.Key challenges include schema normalization, noisy figure/caption extraction, figure-reference alignment, and separating schematics from non-diagram visuals.
  • Metadata acquisition: OpenReview PDFs and metadata are normalized across specified AI/ML venues and years into a Papers relation with bibliographic, semantic, platform, decision, and BibTeX fields.The covered sources include ICLR, NeurIPS, ICML, and TMLR ranges described in the passage.
  • Context extraction: PDFFigures 2.0 extracts figures and captions, while PyMuPDF extracts paragraphs explicitly citing figure numbers to populate figure_context.The context field links each image to the paper’s argumentative flow and provides a retrieval signal beyond captions.
  • Schematic classification: CLIP assigns extracted figures to diagram, plot, photo, or other, then a VLM cascade routes diagram candidates, rescues low-confidence candidates, and resolves specified disagreements.The primary release reserves precision claims for the held-out audit rather than equating cascade acceptance with correctness.
  • Aggregation and views: The denormalized schema joins Papers and Figures so each retrieved exemplar carries title, abstract, caption, context spans, quality labels, and attribution fields.Three released views expose all CLIP-labeled diagrams, high-confidence CLIP diagrams with clip_confidence ≥0.85, and the cascade-filtered accepted-paper release.
  • Quality assurance: 93.67% precision (95% CI 90.11–97.22) is estimated for the 57,100-record release from a stratified blind audit of 500 kept records.The dominant error mode is inclusion of screenshots or photo-like figures, motivating exposed cascade paths and filtering labels.

4 Dataset Statistics and Analysis

DiagramBank’s statistics characterize its scale, filtering views, figure composition, venue coverage, and intended use as an auditable resource for diagram understanding and retrieval.

  • Scale and subset sizes: 452,339 extracted non-table figures comprise plots, diagrams, photos, and other visuals, with 89,422 diagrams classified by the first-stage CLIP classifier.
  • Scale and subset sizes: At clip_confidence ≥0.85, the high-confidence CLIP view provides a simpler precision-oriented filter, while the cascade release includes recovered diagrams initially labeled non-diagrams.
  • Scale and subset sizes: 57,100 cascade-filtered diagrams form the primary release, with 20,516 from ICLR, 11,267 from ICML, 19,655 from NeurIPS, and 5,662 from TMLR.
  • Figure type distribution: Plots account for 65.2% of extracted non-table figures, diagrams 19.8%, and photos 11.5%; diagrams receive lower mean confidence than plots because their styles vary more widely.
  • Visual density and caption characteristics: Visual breadth ranges from 79.8–97.6%, while figure density varies from 4.79 figures per paper at ICLR to 9.22 at TMLR.
  • Visual density and caption characteristics: Venue-specific density and caption differences motivate preserving separate paper-level and figure-level context fields for retrieval and analysis.
  • Resource uses and suggested protocols: The dataset supports diagram-understanding tasks, precision- or coverage-oriented retrieval protocols, literature discovery, and provenance-aware authoring workflows.

5 Example Metadata Uses

DiagramBank’s metadata supports lightweight retrieval and authoring examples, with complementary paper-level and caption-level search rather than a new retrieval method. Its confidence-controlled release and audit guidance make filtering choices explicit for downstream use.

  • Example Metadata Uses: The reference pipeline retrieves schematic exemplars using paper titles, abstracts, and intended captions or descriptions, then inspects layout, grouping, iconography, and terminology.The examples are presented as reference utilities, not as a new retrieval or RAG method.
  • Example Metadata Uses: Three separate indices support coarse domain filtering, paper-level problem and methodology matching, and caption-level matching over modules, components, and relations.The indices use title, abstract, and caption metadata rather than mixing all text into one embedding.
  • Example Metadata Uses: Hierarchical retrieval helps when paper context disambiguates generic captions, while caption-only retrieval can find relevant diagrams from topically different papers.The sanity check characterizes these strategies as complementary rather than one dominating the other.
  • Example Metadata Uses: The release includes records, confidence-controlled views, cascade-path labels, audit materials, indexing code, a manifest, and a dataset card for review and downstream selection.Protocols recommend naming the subset, confidence threshold, cascade paths, and any screenshot or photo filters used in each evaluation.
  • Example Metadata Uses: 94.4% precision is reported for the high-confidence CLIP view at τ = 0.85, while the final release audit estimates 93.67% precision.The CLIP view also reports a 0.49% false-positive rate, and increasing the threshold to 0.90 yields only a small precision gain at greater retention cost.

6 Conclusion

DiagramBank is introduced as a quality-audited dataset of 57,100 scientific schematic diagrams with multi-level document context. Its audits quantify filtering trade-offs and its confidence-controlled views let users choose stricter or broader subsets.

  • 6 Conclusion: 57,100 scientific schematic diagrams are linked to titles, abstracts, captions, in-text figure-reference spans, venue/year metadata, provenance, confidence scores, and cascade labels.The dataset is presented as a resource with multi-level document context.
  • 6 Conclusion: The CLIP and cascade audits quantify precision trade-offs in automated diagram collection, while confidence-controlled views support stricter or broader use.

Limitations

DiagramBank’s automated extraction and precision-oriented release create known scope and coverage boundaries. Its venue coverage and downstream utilities also limit how broadly results should be interpreted.

  • Limitations: Automated PDF parsing, figure classification, and context extraction can produce imperfect captions, incomplete figure-reference spans, and diagram–plot classification errors.
  • Limitations: The precision-oriented primary release may exclude useful lower-confidence diagrams, so higher-coverage users should inspect intermediate CLIP views and apply task-specific filtering.
  • Limitations: Coverage is limited to accessible OpenReview venues and years, which may bias the resource toward recent AI/ML publication norms.
  • Limitations: Retrieval utilities are compact metadata-use sanity checks, and the authoring example is qualitative; downstream claims require task-specific human studies or benchmark protocols.

Ethics Statement

DiagramBank’s use of public scholarly documents requires attention to provenance, redistribution, privacy, bias, misuse, and AI-use disclosure. Its source and venue scope also constrain representativeness and licensing.

  • Ethics Statement: Figures originate from third-party papers and may have copyright or venue-specific license restrictions, so the release distributes extracted images only where permitted.Otherwise it provides metadata, source links, or license-filtered subsets while preserving source identifiers and BibTeX entries.
  • Ethics Statement: The corpus reflects OpenReview AI/ML publication practices and should not be treated as demographically or scientifically representative.
  • Ethics Statement: Generated figures using retrieved exemplars can mislead without disclosure or checking, so users should track provenance, cite source papers, and manually verify factual and visual details.
  • Ethics Statement: The source-document collection does not intentionally target non-public personal data, but publication metadata and figure contents may still contain identifying information.
  • Ethics Statement: Generative AI did not create source records; learned models are used only in the documented classification and example-use pipeline.

A Supplementary Statistics

The supplementary statistics describe corpus composition, temporal and venue coverage, caption patterns, and the denormalized metadata exposed for downstream use.

  • Corpus Statistics: Year-based analyses should account for changing OpenReview venue coverage when interpreting extracted-figure volume.
  • Corpus Statistics: Figure 4 summarizes caption length, extracted-figure volume, and subject-area distributions across years and venues.Caption verbosity varies by venue, volume reflects publication and coverage growth, and taxonomy differences motivate normalized metadata.
  • Metadata Schema: Paper-level fields make records self-contained for retrieval and attribution.
  • Metadata Schema: Figure-level and quality fields expose extraction, filtering, and user-side subset controls.

C Qualitative Authoring Use Case

The qualitative use case shows DiagramBank-conditioned authoring and retrieval as reference workflows: exemplars provide conventions, terminology, grouping choices, and provenance for human revision.

  • Authoring Comparison: Retrieved exemplars provide adjacent papers, visual terminology, grouping choices, and source-level provenance for human revision.
  • Retrieved Exemplars: Other query families retrieve domain-specific conventions, including frozen/trainable iconography, Dreamer terminology, and protein-model notation.
  • Authoring Comparison: DiagramBank-conditioned output is compared with text-only output as an illustrative authoring use case, not a visual-quality benchmark.
  • Authoring Comparison: The text-only authoring panel is identified as a code-assisted reasoning example.
  • Retrieved Exemplars: Figure 6 retrieves provenance-aware examples spanning code-assisted reasoning, code benchmarking, and code-graph modeling.

D Dataset Card / Datasheet and Release Protocols

The dataset card defines DiagramBank’s intended uses, recommends flexible confidence-aware evaluation and user-side quality checks, and specifies licensing, attribution, and takedown responsibilities.

  • Intended Use: DiagramBank supports retrieval of schematic exemplars, future multimodal retrieval evaluations, and exemplar-driven figure authoring.
  • Intended Use: The dataset is not intended for identity inference, author profiling, or producing misleading scientific artifacts.
  • Evaluation Protocols: Future evaluations may use temporal splits, venue holdouts, and confidence-controlled views rather than one fixed benchmark split.
  • User-Side Filtering: Users should report their subset, confidence threshold, and cascade paths, applying task-specific checks to automated extraction when needed.
  • Release Responsibilities: Metadata and code use explicit research licenses, while images are redistributed only when permitted or represented through source links and attribution metadata.
  • Release Responsibilities: Users are asked to cite DiagramBank and original source papers, and maintainers will honor reasonable rights-holder takedown requests.
Loading 2604.20857v2…