SciCap: Generating Captions for Scientific Figures

1 Introduction

• Low-quality captions in published papers can miss opportunities to explain scientific narratives, despite captions’ role in communicating figures’ messages.The paper aims to develop automatic models that generate high-quality captions for scientific figures and charts.Source: §1 · p. 1
• Caption models trained on informative captions could suggest better captions for researchers and improve narration for blind or visually impaired readers.The paper notes that only a few prior works focused on scientific figures, and that a specialized model could help even when the original caption is unhelpful.Source: §1 · p. 1
• SCICAP contains more than two million real-world scientific figures from over 290,000 computer science arXiv papers after several preprocessing steps.The processing included figure-type classification, subfigure identification, text normalization, and caption text selection.Source: §1 · p. 1
• Graph plots, the dominant figure type at 19.2%, are the focus of the paper’s baseline captioning models.The reported experiments indicate both opportunities and steep challenges for scientific-figure caption generation.Source: §1 · p. 1

2 Related Work

• FigCAP pairs synthesized figures with captions, while FigJAM generates caption units such as titles, label names, and minimum or maximum values.The passage also situates these systems alongside earlier figure-content datasets and analysis work.Source: §2 · p. 2
• Data-to-caption approaches use a chart’s source data table and metadata, rather than the figure image, to generate captions.Source: §2 · p. 2
• Prior work generated scientific-figure captions using synthetic images and texts, whose captions may describe features without conveying higher-level insights.The paper contrasts a generic synthetic caption with a human-written caption that highlights meaningful figure content and context.Source: §2 · p. 2

3 Constructing SCICAP Dataset

• The corpus was drawn from 295,028 computer-science and machine-learning arXiv papers published from 2010 to 2020, and extraction produced 2,170,719 figures.The paper collection was frozen on December 22, 2020; source files were not used.Source: §3 · p. 2
• A seven-type classifier labeled graph plots as 19.2% of extracted figures, the most common classified type, and the study focused on graph plots.The classifier’s reported accuracy was 86% over 60,000 samples.Source: §3 · pp. 2–3
• Because SCICAP targets single figures, the pipeline removed figures with subfigures using caption rules and FigureSeparator.FigureSeparator is CNN-based; in the graph-plot collection, the rule stage yielded 352,719 figures and the model reduced that to 133,543.Source: §3 · p. 3
• 416,804 graph plots identified in classification were reduced to 133,543 figures after subfigure filtering; an estimated 32.04% did not contain subfigures.The rule-based stage retained 352,719 plots before FigureSeparator further narrowed the collection.Source: §3 · p. 3
• Caption text was normalized by lowercasing and replacing numbers with [NUM], equations with [EQUATION], and bracketed spans with [BRACKET].The text-normalization passage also says figure numbers were removed and only main caption text was retained.Source: §3 · p. 3
• SCICAP provides First Sentence (133,543 figures), Single-Sentence Caption (94,110 figures), and Caption with No More than 100 Words (131,319 figures) collections; the last includes captions of at most 100 tokens, punctuation included.The Single-Sentence Caption collection contains complete captions only for figures with one-sentence captions.Source: §3 · p. 3
• 23.19 tokens (SD=20.86), 14.05 tokens (SD=8.15), and 22.04 tokens (SD=17.44) were the average sentence lengths for the three collections, respectively.These figures are reported with advanced normalization.Source: §3 · p. 3

3.1 Data Analysis and Quality Measurement

• On 1,926 manually labeled images, graph-plot type classification was described as reliable, and FigureSeparator had exceptionally high precision for identifying plots without subfigures.The labeled sample was used to evaluate the tools in the construction pipeline.Source: §3.1 · pp. 3–4, Table 1 · p. 3

Baseline Model

• The baseline uses a ResNet-101 image encoder and an attention-based LSTM decoder; variants use visual, text, or combined features.Text features are figure titles, legends, and X-Y labels; models were evaluated with BLEU-4 against the corresponding test collection.Source: §4 · p. 4

• Single-Sentence Caption models performed best, though low BLEU-4 scores indicate reliable scientific-figure captioning remains challenging.Source: Table 2 · p. 4, §4 · p. 5

Effects of Text and Vision Features

• Vision-Only and Text-Only features yielded similar performance, while combining them performed slightly worse.Source: Table 3 · p. 5, §4 · p. 5

Effects of Text Normalization

• Experiments found no clear BLEU-4 benefit from text normalization.The authors propose exploring more advanced equation-identification techniques for normalization.Source: §4 · p. 5

• SCICAP contains more than two million scientific figures from over 290,000 arXiv papers.Source: §5 · p. 5
• The image-captioning baselines show both the feasibility and challenges of generating captions for scientific figures.Source: §5 · p. 5
• Future work will explore large pretrained language models and information from papers’ full text to improve caption quality.Source: §5 · p. 5

• The arXiv dataset uses the CC0 1.0 Public Domain Dedication license, which permits remixing, annotation, and publication.Source: Ethical Considerations · p. 6
• Language technologies trained on mainstream English can favor that variety and harm people who use varieties with fewer speakers.Source: Ethical Considerations · p. 6