Source-linked AI summary
SCAFFOLD: A Large-Scale Structured Dataset of Computer Science Research Figures with Diagram QA and Chain-of-Thought Reasoning Traces
Ranjit Raut, Aarav Subedi, Sagun Rai, Sudan Jha
TL;DR
Computer science diagram understanding lacks a public dataset pairing figures with captions, context, questions, answers, and reasoning traces. SCAFFOLD automatically constructs such records from arXiv papers and provides three dataset scales; baseline fine-tuning on SCAFFOLD-12K achieved 99.5% format compliance with moderate answer accuracy, while larger collections remain less thoroughly checked.
Problem
No public dataset pairs computer science diagrams with captions, context, questions, answers, and reasoning traces for training domain-specific vision-language models.
Method
SCAFFOLD automatically extracts figures from arXiv papers, links captions and context, and generates structured question-answer and reasoning records with an AI path and deterministic fallback.
Results
99.5% format compliance was achieved when Qwen2.5-VL-3B-Instruct was fine-tuned on SCAFFOLD-12K, with moderate answer accuracy and stronger performance on numeric questions.
Takeaways & Limitations
SCAFFOLD supports supervised fine-tuning for computer science diagram understanding and chain-of-thought reasoning generation.
Takeaways & Limitations
Only SCAFFOLD-12K was manually spot-checked, while larger collections lacked equivalent independent record-quality checks.
Abstract
from arXiv · showhide
Computer science papers rely heavily on diagrams: architecture drawings, system flowcharts, and pipeline schematics that often carry more information than the text around them. There is currently no public dataset that pairs this specific kind of figure with captions, context, questions, answers, and step-by-step reasoning, which is exactly what is needed to train a vision-language model to understand them. We present \textbf{SCAFFOLD}\footnote{https://github.com/theranjitraut/scaffold}, a large-scale structured dataset of computer science research figures with diagram QA and Chain-of-Thought reasoning traces. This dataset consists of (image, caption, context, question-answer, chain-of-thought) tuples from arXiv computer science papers prepared using layout detection and PDF parsing, with an AI-assisted question-generation step. The resulting large-sized SCAFFOLD-157K dataset spans 3,058 papers with 29,887 figures (157,387 pairs), a medium-sized SCAFFOLD-37K dataset (36,797 pairs), and a small-sized SCAFFOLD-12K dataset (12,000 pairs). We used SCAFFOLD-12K for baseline experiments on Qwen2.5-VL-3B-Instruct.
1 Introduction
SCAFFOLD addresses the lack of datasets for understanding computer science diagrams, whose relationships often require tracing connections and process order rather than recognizing objects alone.
- Computer science diagrams can encode relationships that would otherwise require several paragraphs of explanation.
- Existing vision-language models are rarely trained or evaluated on architecture diagrams, pipelines, and flowcharts.These figures require tracing connections or step order, not just recognizing visual objects.
- SCAFFOLD provides a reproducible pipeline for mining figures, captions, context, questions, answers, and reasoning traces from public research papers.The paper frames this missing data as a barrier to developing domain-specific vision-language models.
2 Related Work
Prior datasets cover synthetic charts, scientific graphs, document images, and infographics, but not figure-level reasoning over computer science system diagrams. SCAFFOLD combines diagram-focused extraction with contextual and reasoning supervision for this under-resourced setting.
- Existing visual question-answering datasets generally target synthetic charts, scientific graphs, or document images rather than computer science architecture diagrams.FigureQA, DVQA, and PlotQA use synthetic charts; SciGraphQA focuses on scientific graphs; DocVQA and InfographicVQA target dense layouts.
- SCAFFOLD isolates individual diagrams, pairs them with captions and surrounding paper context, and supports figure-level reasoning.This structure distinguishes it from datasets designed primarily for text-heavy documents or graphs.
- The paper frames technical diagrams as an under-resourced domain analogous to low-resource languages in NLP.It argues that targeted benchmark construction is needed to make this domain tractable for vision-language systems.
- The extraction pipeline builds on supervised figure-caption separation and combines modern layout detection with multimodal reasoning prompting.It updates the PDFFigures 2.0 precedent using YOLOv8 and incorporates chain-of-thought prompting literature.
3 Dataset Construction
SCAFFOLD extracts and structures figure-level records from arXiv PDFs, links each figure to its caption and contextual sentence, and generates labeled questions and reasoning traces through AI-assisted or deterministic fallback paths.
- Extraction Pipeline: The pipeline processes 3,058 arXiv papers page by page, rendering each page at 150 DPI for layout detection.
- Extraction Pipeline: YOLOv8 detects Picture and Caption regions, while PyMuPDF crops figures directly from source PDFs to preserve image quality.
- Extraction Pipeline: Caption matching uses vertical gap and horizontal bounding-box overlap, and context linking selects the first sentence mentioning the figure number.
- Extraction Pipeline: 29,887 figures were extracted across 3,058 source papers, with incomplete caption or figure-number records flagged rather than silently discarded.
- Generation Pipeline: Each figure-caption-context triple receives AI-generated or deterministic template-based question-answer content, with seven question types.The AI path conditions generation on the image, caption, and referencing sentence; the fallback remains structurally valid when generation fails.
- Data Schema: Records use a shared 14-field schema and are serialized as both inspection-ready JSON and compact model-ready training records.The schema is shared across all three dataset versions, while the compact form removes inspection-only fields and uses the target chat format.
4 Dataset Statistics
SCAFFOLD provides three dataset scales and a structured datasheet, with the smallest collection manually spot-checked and used for baseline training. Its intended use is domain-specific diagram understanding rather than general visual question answering.
- Scale: 29,887 figures from 3,058 papers support the full SCAFFOLD release, while SCAFFOLD-12K was spot-checked and used for baseline training.The larger SCAFFOLD-157K and SCAFFOLD-37K collections demonstrate extraction at scale.
- Question Types: The generation pipeline targets seven question types and favors relationship and component questions to test connections among diagram parts.
- Generation Sources: Each record stores provenance indicating whether questions, answers, and reasoning traces came from the Gemini path or synthetic fallback.Users can filter for Gemini-generated records or use fallback records that are structurally valid but more templated.
- Composition: The dataset pairs each figure with caption, referencing sentence, generated QA, question type, reasoning trace where available, and provenance metadata.
- Collection Process: Instances are generated automatically from arXiv PDFs, with a manual quality review of 1,000 samples rather than individual figure annotation.
- Uses: SCAFFOLD is intended for fine-tuning and evaluating vision-language models on computer science diagram understanding and reasoning generation.It is explicitly not intended for non-CS scientific figures or general-purpose visual question answering.
5 Experiments
The experiments fine-tune Qwen2.5-VL-3B-Instruct on Scaffold-12K and evaluate it on the matching validation split. Results assess dataset usability and pipeline correctness rather than state-of-the-art performance.
- Baseline fine-tuning: Qwen2.5-VL-3B-Instruct was fine-tuned on Scaffold-12K with QLoRA and evaluated on its 2,000-example validation split.The setup used 4-bit quantization and rank-64 LoRA adapters across attention and MLP projection layers.
- Results: 99.5% format compliance was achieved for well-formed <think>/<answer> outputs.The model reached moderate answer accuracy and performed better on numeric than open-ended text questions.
- Benchmarks: The Scaffold-12K baseline uses three reported benchmarks: ChartQA, DocVQA, and AI2D.
6 Limitations and Ethical Considerations
The paper identifies limitations in verification scale, domain coverage, generation dependence, detector generality, reasoning-trace reliability, and licensing. It also discusses third-party API exposure and dataset content assumptions.
- Scale of verification: Only Scaffold-12K was manually spot-checked, while the larger collections lack equivalent independent quality scrutiny.
- Domain coverage: Source papers cover limited computer science subfields, potentially underrepresenting diagram styles from less-represented areas.
- Generation dependency: Question, answer, and reasoning quality depends substantially on Gemini, whose fallback generates simpler and more uniform questions.
- Layout detector generality: The DocLayNet-trained detector is not specialized for computer science diagrams, and unusual-layout accuracy has not been formally measured.
- Reasoning trace reliability: Model- or template-generated reasoning traces lack formal large-scale human evaluation for factual correctness and reasoning quality.
- Copyright and source material: Redistribution of extracted figures must follow each arXiv paper’s license rather than assuming uniform terms.The annotations, metadata, organization, and generated traces use CC BY-NC 4.0, while extracted figures inherit source-paper licensing.
7 Data Availability Statement
The project intends to release its extraction, matching, and generation pipeline as open-source software alongside trained-model checkpoints. Dataset hosting and release metadata are described separately.
- Release plans: The extraction, matching, and generation code is intended for open-source release with associated trained-model checkpoints.
- License: Annotations, metadata, dataset organization, and generated reasoning traces are released under CC BY-NC 4.0.Extracted figures retain the licenses of their individual arXiv source papers, with a per-paper license manifest planned.
- Hosting and identification: The dataset is hosted on Kaggle and Hugging Face Datasets, with a DOI planned through the hosting repository.