Source-linked AI summary
Extracting Scientific Figures with Distantly Supervised Neural Networks
Noah Siegel, Nicholas Lourie, Russell Power, Waleed Ammar
TL;DR
Scientific figure extraction is difficult to develop for PDF papers because large labeled datasets are lacking. The paper induces labels by aligning markup-specified figures and tables with PDF bounding boxes, then trains a neural detector. DeepFigures performs better on PubMed but worse on CS-Large than PDFFigures 2.0, and has been used to extract figures in 13 million papers.
Problem
Large labeled datasets for scientific figure extraction are lacking, despite the importance of figures and tables in scientific documents.
Source: §1 · p. 1Method
The authors align figures and tables specified in document markup with bounding boxes in corresponding PDFs to induce labels for training figure-extraction models.
Source: §3 · p. 3Results
DeepFigures underperforms PDFFigures 2.0 by 3 F1 points on CS-Large but achieves a 17 point improvement on PubMed.
Source: §5 · p. 8Takeaways & Limitations
DeepFigures has been used to extract figures in 13 million papers, demonstrating scalability and robustness across a variety of domains.
Source: §7 · p. 9Full end-to-end extraction, including caption detection and matching, remains future work.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Non-textual components such as charts, diagrams and tables provide key information in many scientific documents, but the lack of large labeled datasets has impeded the development of data-driven methods for scientific figure extraction. In this paper, we induce high-quality training labels for the task of figure extraction in a large number of scientific documents, with no human intervention. To accomplish this we leverage the auxiliary data provided in two large web collections of scientific documents (arXiv and PubMed) to locate figures and their associated captions in the rasterized PDF. We share the resulting dataset of over 5.5 million induced labels---4,000 times larger than the previous largest figure extraction dataset---with an average precision of 96.8%, to enable the development of modern data-driven methods for this task. We use this dataset to train a deep neural network for end-to-end figure detection, yielding a model that can be more easily extended to new domains compared to previous work. The model was successfully deployed in Semantic Scholar, a large-scale academic search engine, and used to extract figures in 13 million scientific documents.
1 INTRODUCTION
The paper addresses the shortage of scalable figure-extraction labels and the limited domain reach of earlier rule-based methods. It induces labels from scientific-document collections and uses them to train a neural detector, with the resulting data and tools released for further research.
1 INTRODUCTION
- Earlier rule-based methods focused on limited domains and did not generalize well because academic fields vary in stylistic conventions.Prior work focused on selected AI conferences or computer science papers.Source: §1 · p. 1
- 5.5 million induced labels achieve 96.8% average precision and are three orders of magnitude larger than human-labeled figure-extraction datasets.Labels are induced without human intervention from auxiliary data in arXiv and PubMed.Source: §1 · pp. 1–2
- Throughout the paper, “figure extraction” includes extracting both figures and tables.Source: §1 · p. 1
- DeepFigures uses a neural network trained on the induced dataset and generalizes better across scientific domains than prior rule-based techniques.The model is built on ResNet-101 and is deployed in Semantic Scholar.Source: §1 · p. 2
- The paper releases its figure-extraction data, tool, and dataset-generation code to support further research in graphical information understanding.Source: §1 · p. 2
2 RELATED WORK
The related work covers scientific-figure applications and extraction, followed by neural computer-vision methods underlying DeepFigures. It frames domain generalization as a challenge and reviews ResNet-101 and OverFeat as relevant model components.
2 RELATED WORK
- The related-work section groups prior research into scientific-figure extraction and understanding, and neural models used for computer vision.Source: §2 · p. 2
2.1 Scientific Figures
- Prior work uses scientific figures for applications such as plot-data extraction, search, and information extraction, while extraction methods have focused on limited domains.The paper also situates its approach in distant supervision: projecting labels from an existing resource to unlabeled instances without the strong assumption described for an earlier example.Source: §2.1 · p. 2
2.2 Neural Models for Computer Vision
- Convolutional neural networks learn image representations from data, contrasting with earlier approaches that relied on hand-engineered features.The section reviews neural models for computer vision as building blocks for DeepFigures.Source: §2.2.1 · p. 2, §2.2 · p. 2
2.2.1 ResNet-101.
- ResNet identity connections provide a path for gradients to reach earlier layers, supporting the training of deeper networks.ResNet-101 is a 101-layer feature-extraction architecture composed of bottleneck units and identity connections.Source: §2.2.1 · pp. 2–3
2.2.2 OverFeat.
- Object detection predicts bounding boxes, and OverFeat performs bounding-box regression across image regions before merging multiple predictions.Its fully convolutional model runs on cropped regions centered on a uniform grid; grid-based processing shares work across overlapping regions.Source: §2.2.2 · p. 3
3 INDUCING FIGURE EXTRACTION LABELS
The paper induces figure and table labels by aligning markup with rendered PDFs, using separate approaches for LaTeX and XML. It compares induced labels with manual annotation and evaluates their accuracy against human judgments.
3 INDUCING FIGURE EXTRACTION LABELS
- The method aligns markup-specified figures and tables with PDF bounding boxes, using their coordinates as labeled data.The paper describes this dataset as useful for training statistical models, including deep neural networks.Source: §3 · p. 3
3.1 Aligning Figures in LaTeX Documents
- For LaTeX, colored borders and pixel differences locate figure and table boxes; colored names and captions help identify text regions before captions are excluded.The box adjustment handles captions above tables or beside figures.Source: §3.1 · pp. 3–4, Figure 1 · p. 4
3.2 Aligning Figures in XML Documents
- The LaTeX alignment route cannot yield large labeled datasets in fields where LaTeX is less popular, including medical and life sciences.XML markup is available for some publishers’ papers, including the PubMed Central Open Access Subset.Source: §3.2 · p. 4
- For XML figures, caption text is matched to PDF text using minimum edit distance, then the corresponding image is template-matched against the rendered page.The method uses 45 scales; papers are excluded when any figure match scores below 0.8.Source: §3.2 · pp. 4–5
- For XML tables, bag-of-words similarity locates the PDF text interval because PDF-extracted and XML cell orders can differ.The search uses a word-difference counter to find the interval with the lowest distance to the XML table.Source: §3.2 · p. 5
3.3 Comparison to Manual Annotation
- Manual annotation is slow, expensive, and difficult to scale, while the induced datasets are described as scalable and accurate by comparison.The comparison includes the previously largest figure-extraction dataset, CS-Large.Source: §3.3 · p. 5
- Sampled induced labels achieved F1 scores from 93.9% to 100% under checks requiring correct figure corners and caption quadrants.The evaluation covered 150 pages in 61 LaTeX-dataset papers and 106 pages in 86 XML-dataset papers.Source: Table 2 · p. 6, §3.3 · p. 6
4 THE DEEPFIGURES MODEL
DeepFigures detects figures in rendered PDF pages using a fully convolutional ResNet-101 detector. A separate pipeline extracts captions and pairs them with detected figures by minimizing the distance between their centers.
4 THE DEEPFIGURES MODEL
- Unlike OverFeat, DeepFigures trains the full ResNet-101 network end-to-end, learning features for localization rather than fixing classification-trained embedding weights.The authors also use ResNet-101 instead of OverFeat’s shallower five-layer feature grid.Source: §4 · p. 6
- The model processes each page as a 20x15 grid of 1024-dimensional ResNet-101 embeddings and predicts a bounding box and confidence score at each cell.It produces 300 candidate boxes, then applies non-maximum suppression and a confidence threshold.Source: §4 · p. 6, Figure 3 · p. 7
- The pipeline detects figure boxes with a neural network but extracts captions separately using rules that find paragraphs beginning with figure or table labels.The authors report that training a neural network to detect captions reduced performance, citing their small height, weak visual distinctiveness, and a strong existing caption detector.Source: §4 · pp. 6–7
- Figures are paired with captions by minimizing the total Euclidean distance between their box centers, using the Hungarian algorithm for the assignment.When the two counts differ, the algorithm pairs the smaller of the figure and caption sets.Source: §4 · pp. 6–7
- At test time, the model uses a 50% confidence threshold, which can be adjusted to favor precision or recall.Redundant boxes are removed through non-maximum suppression.Source: §4 · p. 7
5 EXPERIMENTS
DeepFigures is compared with PDFFigures 2.0 on human-annotated computer-science and PubMed data. It scores lower on CS-Large but higher on PubMed, with evaluation based on figure and caption matching criteria.
5 EXPERIMENTS
- The model is trained on induced figure and table boxes and evaluated on CS-Large and a new expert-annotated PubMed set of 289 figures and 124 tables from 104 papers.The training data includes 4,095,622 induced figures and 1,431,820 induced tables; evaluation is entirely on human-labeled data.Source: §5 · p. 7
- Evaluation uses human-annotated figures; boxes count as correct at IOU above .80, with an additional text-match option for caption boxes.Unlike the prior evaluation, figure matching does not require annotations to include figure numbers; Hungarian assignment matches predicted and true figures page by page.Source: §5 · p. 8
- DeepFigures scores 3 F1 points lower on CS-Large and 17 points higher on PubMed than PDFFigures 2.0.The authors note that PDFFigures 2.0 was tuned specifically for the CS-Large test set.Source: §5 · p. 8, Table 3 · p. 8
- DeepFigures rejects graphical-header false positives that PDFFigures 2.0 often produces, despite those headers being visually distinct from actual figures.The authors connect its cross-domain robustness to training without human annotation or domain-specific feature engineering.Source: §5 · p. 8
6 EXTRACTING FIGURES IN PRODUCTION
Semantic Scholar deployed DeepFigures to extract and serve figures for eligible papers at large scale. The production system distributes extraction across workers and scales horizontally.
6 EXTRACTING FIGURES IN PRODUCTION
- 96% of 13 million eligible papers were successfully extracted, with an average of 4.5 figures or tables indexed per paper.Semantic Scholar indexed more than 30 million papers, but extraction was limited to papers that allowed it and had a PDF.Source: §6 · p. 8
- A job queue distributes papers to workers that retrieve PDFs, run DeepFigures locally, crop detected figures, and upload results and images to S3.The stored results can be picked up by other processing systems for transformations or indexing.Source: §6 · pp. 8–9
- Semantic Scholar post-processing transforms extraction results, indexes figure metadata in Elasticsearch, and serves images from S3.Figures attached to a paper can be retrieved from Elasticsearch and their corresponding images served directly from S3.Source: §6 · p. 9
- The deployed system uses GPU-capable AWS instances with eight worker processes to balance GPU, CPU, and network demands.The instance type is configurable, and hardware experimentation helped find a high-throughput setup at reasonable cost.Source: §6 · p. 9
- At full speed, the system processes about 15 papers per minute per machine, with average processing time around 30 seconds per paper.A 12-page PDF takes about 45 seconds for the full pipeline, from queueing through rendering and extraction.Source: §6 · p. 9
7 CONCLUSION
The paper contributes a large, high-quality induced-label dataset and a neural figure-extraction model trained on it. The model was used at scale in an academic search engine, while full end-to-end detection and caption matching remain future work.
7 CONCLUSION
- The paper contributes 5.5 million high-accuracy induced labels to support more advanced figure-extraction methods.The conclusion says DeepFigures was trained on the induced dataset and used to extract figures in 13 million papers.Source: §7 · p. 9
- DeepFigures, trained on the induced dataset, extracted figures in 13 million papers in a large-scale academic search engine.The paper presents this deployment as demonstrating scalability and robustness across domains.Source: §7 · p. 9
- Future work includes end-to-end figure extraction that detects figures and matches their captions.The paper also suggests using PDF text and image locations and extending data generation to titles, authors, and sections.Source: §7 · p. 9