PaperlayerView exact revision on arXiv

Source-linked AI summary

DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context

Ling Yue, Tingwen Zhang, Jiaying Wang, Zhen Xu, Shaowu Pan

arXiv:2604.20857v2cs.IRcs.AI

TL;DR

Existing scientific-figure resources often mix schematic diagrams with other visual materials and omit the document context needed to interpret them. DiagramBank addresses this gap with a quality-audited, context-rich dataset and reports 93.67% precision for its final release.

  • Problem

    Existing scientific-figure resources often mix schematics with plots, photos, tables, or screenshots and omit body-text context explaining figure use.

    Source: §1 · p. 1

  • Method

    DiagramBank combines automated PDF parsing, figure classification, context extraction, and a VLM cascade to release diagrams with multi-level metadata, provenance, labels, and confidence views.

    Source: §1 · p. 1, §3 · p. 3, §4.5 · p. 7

  • Results

    93.67% precision was estimated for the final release, while the dataset contains 57,100 cascade-filtered scientific schematic diagrams.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    The released records support scientific-document understanding, diagram retrieval, corpus analysis, and benchmark-oriented protocols with auditable provenance and filtering choices.

    Source: §4.5 · p. 7, §6 · p. 8

    The precision-oriented release may exclude useful lower-confidence diagrams, coverage is limited to accessible OpenReview venues and years, and downstream utilities remain qualitative sanity checks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Scientific papers use schematic diagrams to communicate methods, workflows, and system structure, yet existing scientific-figure corpora often mix them with plots, screenshots, and photographs and rarely preserve document context. We introduce DiagramBank, a quality-audited dataset of 57,100 schematic diagrams curated from OpenReview-hosted AI/ML venues. Each record links a diagram image to its paper title, abstract, figure caption, in-text figure-reference spans, venue/year metadata, provenance fields, and filtering labels. DiagramBank is a reusable resource for scientific-document understanding, diagram retrieval, corpus analysis, and future benchmark construction. We describe its extraction and cascade-filtering pipeline, release schema, confidence-controlled views, dataset card, and indexing utilities. A manual blind audit of the released cascade-filtered records estimates 93.67% precision, and a separate CLIP threshold analysis characterizes the precision--coverage trade-off for simpler filtering views. We further provide lightweight metadata-indexing and authoring examples to illustrate downstream protocols without treating these utilities as standalone methods. The code is public at: https://github.com/csml-rpi/DiagramBank.

1 Introduction

DiagramBank addresses the lack of diagram-specific scientific-figure resources that preserve document context and filtering provenance. It releases 57,100 quality-audited schematic diagrams with linked paper, figure, and attribution metadata.

Motivation

  • Schematic diagrams encode objects, modules, information flow, layout, arrows, labels, grouping, and community-specific notation.Their structure motivates treating diagram understanding as distinct from ordinary plot or natural-image understanding.Source: §1 · p. 1

Research gap

  • Existing scientific-figure resources often mix schematics with plots, photos, tables, or screenshots and omit explanatory body-text passages.DiagramBank is designed to preserve paper-level and figure-level context, filtering uncertainty, and provenance for audit and attribution.Source: §1 · p. 1

Contribution

  • DiagramBank contains 57,100 cascade-filtered scientific schematic diagrams from OpenReview-hosted AI/ML venues.Each record couples an image with title, abstract, caption, in-text figure-reference spans, venue/year metadata, quality-control labels, and attribution fields.Source: §1 · p. 1
  • The release provides a diagram-specific corpus, measured filtering quality, confidence views, provenance, a dataset card, indexing code, and reporting guidance.The high-confidence CLIP view reaches 94.4% precision at τ = 0.85, while the final release reaches 93.67% precision.Source: §1 · p. 1

2 Related Work and Resource Positioning

DiagramBank is positioned as a diagram-specific, auditable resource that fills gaps left by heterogeneous figure corpora and supports context-aware retrieval and evaluation. It supplies multi-level evidence rather than introducing a new downstream algorithm.

Existing resources

  • Prior resources support captioning, figure classification, summarization, or visual-summary selection but often mix diagrams with other figure types.Examples include SciCap, DocFigure, ACL-Fig, and representative-figure ranking work.Source: §2 · p. 2

Emerging need

  • Automated authoring and diagram-generation systems increase demand for scientific-figure resources that can support grounded workflows.The cited systems include AI Scientist, autonomous paper generation, Paper-Banana, AutoFigure, DiagrammerGPT, and Sci-Doc2Diagrammer.Source: §2 · p. 2

Resource positioning

  • DiagramBank preserves diagram retrieval units, multi-granularity textual evidence, paper metadata, and measured filtering quality for auditable evaluations.It is explicitly positioned as neither a benchmark leaderboard nor a new retrieval algorithm.Source: §2 · p. 2

3 Dataset Construction and Quality Assurance

DiagramBank constructs its resource through normalized metadata and figure-context extraction, followed by CLIP routing and a VLM cascade for schematic classification. Complementary blind audits quantify both confidence-view and final-release precision.

Pipeline

  • The four-stage pipeline comprises metadata acquisition, content extraction, schematic classification, and relational aggregation.Key challenges include schema normalization, noisy figure/caption extraction, figure-reference alignment, and separating schematics from non-diagram visuals.Source: §3 · p. 2

Metadata acquisition

  • OpenReview PDFs and metadata are normalized across specified AI/ML venues and years into a Papers relation with bibliographic, semantic, platform, decision, and BibTeX fields.The covered sources include ICLR, NeurIPS, ICML, and TMLR ranges described in the passage.Source: §3 · p. 3

Context extraction

  • PDFFigures 2.0 extracts figures and captions, while PyMuPDF extracts paragraphs explicitly citing figure numbers to populate figure_context.The context field links each image to the paper’s argumentative flow and provides a retrieval signal beyond captions.Source: §3 · p. 3

Schematic classification

  • CLIP assigns extracted figures to diagram, plot, photo, or other, then a VLM cascade routes diagram candidates, rescues low-confidence candidates, and resolves specified disagreements.The primary release reserves precision claims for the held-out audit rather than equating cascade acceptance with correctness.Source: §3 · p. 3

Aggregation and views

  • The denormalized schema joins Papers and Figures so each retrieved exemplar carries title, abstract, caption, context spans, quality labels, and attribution fields.Three released views expose all CLIP-labeled diagrams, high-confidence CLIP diagrams with clip_confidence ≥0.85, and the cascade-filtered accepted-paper release.Source: §3 · p. 3

Quality assurance

  • 93.67% precision (95% CI 90.11–97.22) is estimated for the 57,100-record release from a stratified blind audit of 500 kept records.The dominant error mode is inclusion of screenshots or photo-like figures, motivating exposed cascade paths and filtering labels.Source: §3 · p. 4

4 Dataset Statistics and Analysis

DiagramBank’s statistics characterize its scale, filtering views, figure composition, venue coverage, and intended use as an auditable resource for diagram understanding and retrieval.

Scale and subset sizes

  • 452,339 extracted non-table figures comprise plots, diagrams, photos, and other visuals, with 89,422 diagrams classified by the first-stage CLIP classifier.Source: §4.1 · p. 4
  • At clip_confidence ≥0.85, the high-confidence CLIP view provides a simpler precision-oriented filter, while the cascade release includes recovered diagrams initially labeled non-diagrams.Source: §4.1 · pp. 4–5
  • 57,100 cascade-filtered diagrams form the primary release, with 20,516 from ICLR, 11,267 from ICML, 19,655 from NeurIPS, and 5,662 from TMLR.Source: §4.1 · p. 4

Figure type distribution

  • Plots account for 65.2% of extracted non-table figures, diagrams 19.8%, and photos 11.5%; diagrams receive lower mean confidence than plots because their styles vary more widely.Source: §4.2 · p. 5

Visual density and caption characteristics

  • Visual breadth ranges from 79.8–97.6%, while figure density varies from 4.79 figures per paper at ICLR to 9.22 at TMLR.Source: §4.3 · p. 5
  • Venue-specific density and caption differences motivate preserving separate paper-level and figure-level context fields for retrieval and analysis.Source: §4.3 · p. 5

Resource uses and suggested protocols

  • The dataset supports diagram-understanding tasks, precision- or coverage-oriented retrieval protocols, literature discovery, and provenance-aware authoring workflows.Source: §4.5 · pp. 6–7

5 Example Metadata Uses

DiagramBank’s metadata supports lightweight retrieval and authoring examples, with complementary paper-level and caption-level search rather than a new retrieval method. Its confidence-controlled release and audit guidance make filtering choices explicit for downstream use.

Example Metadata Uses

  • The reference pipeline retrieves schematic exemplars using paper titles, abstracts, and intended captions or descriptions, then inspects layout, grouping, iconography, and terminology.The examples are presented as reference utilities, not as a new retrieval or RAG method.Source: §5 · p. 7
  • Three separate indices support coarse domain filtering, paper-level problem and methodology matching, and caption-level matching over modules, components, and relations.The indices use title, abstract, and caption metadata rather than mixing all text into one embedding.Source: §5 · p. 7
  • Hierarchical retrieval helps when paper context disambiguates generic captions, while caption-only retrieval can find relevant diagrams from topically different papers.The sanity check characterizes these strategies as complementary rather than one dominating the other.Source: §5 · p. 7, Table 5 · p. 7
  • The release includes records, confidence-controlled views, cascade-path labels, audit materials, indexing code, a manifest, and a dataset card for review and downstream selection.Protocols recommend naming the subset, confidence threshold, cascade paths, and any screenshot or photo filters used in each evaluation.Source: §5 · p. 8
  • 94.4% precision is reported for the high-confidence CLIP view at τ = 0.85, while the final release audit estimates 93.67% precision.The CLIP view also reports a 0.49% false-positive rate, and increasing the threshold to 0.90 yields only a small precision gain at greater retention cost.Source: §5 · p. 8

6 Conclusion

DiagramBank is introduced as a quality-audited dataset of 57,100 scientific schematic diagrams with multi-level document context. Its audits quantify filtering trade-offs and its confidence-controlled views let users choose stricter or broader subsets.

6 Conclusion

  • 57,100 scientific schematic diagrams are linked to titles, abstracts, captions, in-text figure-reference spans, venue/year metadata, provenance, confidence scores, and cascade labels.The dataset is presented as a resource with multi-level document context.Source: §6 · p. 8
  • The CLIP and cascade audits quantify precision trade-offs in automated diagram collection, while confidence-controlled views support stricter or broader use.Source: §6 · p. 8

Limitations

DiagramBank’s automated extraction and precision-oriented release create known scope and coverage boundaries. Its venue coverage and downstream utilities also limit how broadly results should be interpreted.

Limitations

  • Automated PDF parsing, figure classification, and context extraction can produce imperfect captions, incomplete figure-reference spans, and diagram–plot classification errors.Source: Limitations · p. 9
  • The precision-oriented primary release may exclude useful lower-confidence diagrams, so higher-coverage users should inspect intermediate CLIP views and apply task-specific filtering.Source: Limitations · p. 9
  • Coverage is limited to accessible OpenReview venues and years, which may bias the resource toward recent AI/ML publication norms.Source: Limitations · p. 9
  • Retrieval utilities are compact metadata-use sanity checks, and the authoring example is qualitative; downstream claims require task-specific human studies or benchmark protocols.Source: Limitations · p. 9

Ethics Statement

DiagramBank’s use of public scholarly documents requires attention to provenance, redistribution, privacy, bias, misuse, and AI-use disclosure. Its source and venue scope also constrain representativeness and licensing.

Ethics Statement

  • Figures originate from third-party papers and may have copyright or venue-specific license restrictions, so the release distributes extracted images only where permitted.Otherwise it provides metadata, source links, or license-filtered subsets while preserving source identifiers and BibTeX entries.Source: Ethics Statement · p. 9
  • The corpus reflects OpenReview AI/ML publication practices and should not be treated as demographically or scientifically representative.Source: Ethics Statement · p. 9
  • Generated figures using retrieved exemplars can mislead without disclosure or checking, so users should track provenance, cite source papers, and manually verify factual and visual details.Source: Ethics Statement · p. 9
  • The source-document collection does not intentionally target non-public personal data, but publication metadata and figure contents may still contain identifying information.Source: Ethics Statement · p. 9
  • Generative AI did not create source records; learned models are used only in the documented classification and example-use pipeline.Source: Ethics Statement · p. 9

A Supplementary Statistics

The supplementary statistics describe corpus composition, temporal and venue coverage, caption patterns, and the denormalized metadata exposed for downstream use.

Corpus Statistics

  • Year-based analyses should account for changing OpenReview venue coverage when interpreting extracted-figure volume.Source: A Supplementary Statistics · p. 11
  • Figure 4 summarizes caption length, extracted-figure volume, and subject-area distributions across years and venues.Caption verbosity varies by venue, volume reflects publication and coverage growth, and taxonomy differences motivate normalized metadata.Source: Figure 4 · p. 11

Metadata Schema

C Qualitative Authoring Use Case

The qualitative use case shows DiagramBank-conditioned authoring and retrieval as reference workflows: exemplars provide conventions, terminology, grouping choices, and provenance for human revision.

Authoring Comparison

Retrieved Exemplars

Authoring Comparison

  • DiagramBank-conditioned output is compared with text-only output as an illustrative authoring use case, not a visual-quality benchmark.Source: Figure 5 · p. 12
  • The text-only authoring panel is identified as a code-assisted reasoning example.Source: C Qualitative Authoring Use Case · p. 12

Retrieved Exemplars

  • Figure 6 retrieves provenance-aware examples spanning code-assisted reasoning, code benchmarking, and code-graph modeling.Source: Figure 6 · p. 12

D Dataset Card / Datasheet and Release Protocols

The dataset card defines DiagramBank’s intended uses, recommends flexible confidence-aware evaluation and user-side quality checks, and specifies licensing, attribution, and takedown responsibilities.

Intended Use

Evaluation Protocols

User-Side Filtering

Release Responsibilities

Related papers

Loading 2604.20857v2…