Source-linked AI summary
On the Use of ArXiv as a Dataset
Colin B. Clement, Matthew Bierbaum, Kevin P. O'Keeffe, Alexander A. Alemi
TL;DR
The paper asks how arXiv’s multimodal, relational data can become a standardized benchmark for next-generation models. It provides an open-source pipeline for collecting and normalizing metadata, full text, and co-citations, then demonstrates baseline classification. The resulting resources include 1.37 million extracted text documents, an approximately 11 billion-word corpus, and a large co-citation network, supporting further relational modeling tasks.
Problem
Prior arXiv studies used different data subsets, limiting direct comparison; the paper seeks standardized access to a large multimodal, relational benchmark.
Method
The paper provides an open-source pipeline that collects metadata, downloads PDFs, converts them to plaintext, extracts co-citations, and normalizes author strings.
Results
The pipeline yields 1.37 million raw text documents, approximately 11 billion words, and a co-citation network with O(10^6) nodes.
Takeaways & Limitations
The standardized arXiv resources support category classification and motivate relational models for link prediction, summarization, segmentation, and topic modeling.
Abstract
from arXiv · showhide
The arXiv has collected 1.5 million pre-print articles over 28 years, hosting literature from scientific fields including Physics, Mathematics, and Computer Science. Each pre-print features text, figures, authors, citations, categories, and other metadata. These rich, multi-modal features, combined with the natural graph structure---created by citation, affiliation, and co-authorship---makes the arXiv an exciting candidate for benchmarking next-generation models. Here we take the first necessary steps toward this goal, by providing a pipeline which standardizes and simplifies access to the arXiv's publicly available data. We use this pipeline to extract and analyze a 6.7 million edge citation graph, with an 11 billion word corpus of full-text research articles. We present some baseline classification results, and motivate application of more exciting generative graph models.
1 INTRODUCTION
The arXiv combines large-scale multimodal article data with citation, authorship, and category graphs, making it a candidate benchmark for relational modeling. This paper addresses fragmented prior uses by standardizing access to the public data.
- Its multimodal features and complex relational structure could support benchmarking models that operate on graph-structured data.
- The arXiv contains over 1.5 million articles with full text, metadata, and internal co-citations across several scientific communities.
- Prior studies used different arXiv subsets for link prediction, network modeling, text labeling, and document similarity.
- The paper provides an open-source pipeline to standardize, simplify, and normalize access, enabling more directly comparable benchmark results.
2 DATASET
The dataset pipeline collects arXiv metadata and makes its heterogeneous article fields accessible for large-scale analysis. Metadata includes identifiers, authors, titles, abstracts, versions, categories, and optional publication identifiers.
- The pipeline collects arXiv metadata through the Open Archive Initiative and downloads bulk PDFs from arXiv.
- As of 2019-03-01, metadata for 1,506,500 articles was available, including identifiers, submitters, authors, titles, abstracts, versions, and categories.
- Authors are provided as a single string, and the repository includes a suggested script for splitting and normalizing author names.
2.2 FULL TEXT
The paper extracts a large plaintext corpus and constructs an arXiv co-citation network from article text. These resources provide scale, technical content, graph structure, and baseline modeling opportunities, while extracted text remains imperfectly clean.
- FULL TEXT: The pipeline converts bulk-downloaded PDFs to plaintext with a standard PDF-to-text converter.
- FULL TEXT: 1.37 million raw text documents can currently be extracted from arXiv PDFs using the pipeline.
- FULL TEXT: Approximately 64 GB of extracted text contains 11 billion words, offering a large technical corpus that also supports bulk study of mathematical formulae.
- CO-CITATIONS: The co-citation network is extracted by searching article text for valid arXiv identifiers and is distributed as a compressed binary.
- CO-CITATIONS: The network contains O(10^6) nodes, 62% in its largest weakly connected component, and 31% fully isolated nodes.
- CO-CITATIONS: Baseline category classification trains logistic regression on 1.2 million articles using Universal Sentence Encoder representations of titles and abstracts.
3 CONCLUSION
The paper presents the arXiv as a large, richly annotated relational dataset and releases tools to standardize access, enabling broader model development and evaluation.
- 3 CONCLUSION: The open-source repository makes arXiv data easier to download and standardize for relational modeling.The dataset combines authorship, categories, abstracts, co-citations, and full text.
- 3 CONCLUSION: Table 3 evaluates baseline classification on a 390k-article holdout set after training logistic regression on 1.2 million articles.Features include title, abstract, full text, and co-citation representations.
- 3 CONCLUSION: The benchmark supports progress on category inference and tasks including link prediction, automatic summary generation, text segmentation, and time-varying topic modeling.The authors connect these opportunities to models that encode arXiv graph structure.
A LOGISTIC REGRESSION ARTICLE CLASSIFICATION BASELINE
The baseline classifies arXiv articles using supervised category labels and features derived from article text and citation structure.
- A LOGISTIC REGRESSION ARTICLE CLASSIFICATION BASELINE: ArXiv articles have one primary category assigned by the submitter and confirmed by the moderation system.The frozen dataset contains 175 possible categories, and baselines use titles, abstracts, full text, and co-citation features.
A.1 BUILDING FEATURES
The feature-construction procedure embeds article text and combines category information with first-order citation relationships while preventing test-set leakage.
- A.1 BUILDING FEATURES: 1,506,500 articles have metadata, while 1,357,536 have full text; each text field is mapped to 512 dimensions, with zeros for missing full text.Title, abstract, and full-text strings are embedded using the pretrained Universal Sentence Encoder.
- A.1 BUILDING FEATURES: The co-citation graph is represented through an N × N matrix whose entries indicate whether article i cites article j or vice versa.This matrix supplies the graph relationship used for citation-derived features.
- A.1 BUILDING FEATURES: Citations from training articles to test articles are omitted to prevent test-set leakage, while test-to-training citations are retained.
- A.1 BUILDING FEATURES: The category matrix is an N × L one-hot representation indicating whether article j belongs to category l.
- A.1 BUILDING FEATURES: The co-citation feature matrix is MC, using only first-order citation relationships in this paper.Higher-order terms such as M^2C and M^3C are described but not used.
A.2 TRAINING
The training procedure concatenates selected text and co-citation features, fits an SGD classifier, and evaluates it on a held-out article set.
- A.2 TRAINING: The model concatenates selected title, abstract, full-text, and co-citation vectors before fitting an SGD classifier.The classifier uses logistic loss, tolerance 1e-6, 50 maximum iterations, and alpha=1e-7.
- A.2 TRAINING: 1,200,000 articles are used for training and 306,500 for testing after randomly shuffling the dataset.The evaluation reports classification accuracy and model perplexity.