Source-linked AI summary
A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics
Martin Gerlach, Francesc Font-Clos
TL;DR
Studies using Project Gutenberg have often relied on small, potentially biased book subsets and inconsistent preprocessing, raising reproducibility concerns. The paper presents SPGC, a processed full-size corpus with metadata and multiple data granularities. It characterizes the collection and finds bookshelf labels more evenly distributed and more often unique to a book than subject labels.
Problem
PG studies often use small, potentially biased book subsets or inconsistent, insufficiently documented preprocessing, raising concerns about reproducibility.
Source: §I · p. 1Method
SPGC automatically downloads, filters, and processes PG books, combines metadata sources, and provides raw text, token time series, and word counts.
Source: §I · p. 2, §II · p. 2Results
SPGC covers more than 50,000 books in more than 20 languages, and up to 85% of books have a unique bookshelf label compared with roughly 50% having two or more subject labels.
Source: §IV · p. 3, §VI · p. 5Takeaways & Limitations
The authors present SPGC as a reproducible scientific resource and a starting point for more rigorous use of PG in quantitative language studies.
Source: §VI · pp. 5–6
Abstract
from arXiv · showhide
The use of Project Gutenberg (PG) as a text corpus has been extremely popular in statistical analysis of language for more than 25 years. However, in contrast to other major linguistic datasets of similar importance, no consensual full version of PG exists to date. In fact, most PG studies so far either consider only a small number of manually selected books, leading to potential biased subsets, or employ vastly different pre-processing strategies (often specified in insufficient details), raising concerns regarding the reproducibility of published results. In order to address these shortcomings, here we present the Standardized Project Gutenberg Corpus (SPGC), an open science approach to a curated version of the complete PG data containing more than 50,000 books and more than $3 \times 10^9$ word-tokens. Using different sources of annotated metadata, we not only provide a broad characterization of the content of PG, but also show different examples highlighting the potential of SPGC for investigating language variability across time, subjects, and authors. We publish our methodology in detail, the code to download and process the data, as well as the obtained corpus itself on 3 different levels of granularity (raw text, timeseries of word tokens, and counts of words). In this way, we provide a reproducible, pre-processed, full-size version of Project Gutenberg as a new scientific resource for corpus linguistics, natural language processing, and information retrieval.
I. INTRODUCTION
Project Gutenberg (PG) has become a major source for quantitative language research, but studies have used small, potentially biased subsets and inconsistent processing. The paper introduces SPGC as a standardized version of the full PG collection, with metadata and reproducible corpus files.
I. INTRODUCTION
- PG supports quantitative studies of language, including work on statistical properties, language change, information content, and syntax.The introduction also describes PG as indispensable to quantitative linguistics.Source: §I · p. 1
- Many studies use small, repeatedly selected subsets of PG, while differing or underreported processing methods can produce inconsistent datasets.These practices raise concerns about replicability and generalizability.Source: §I · p. 1
- SPGC provides a standardized, complete PG corpus with metadata, processing code, and raw text, token time series, and word counts.It can be recreated and updated as books are added to PG, with a static SPGC-2018-07-18 version for reproducibility.Source: §I · p. 2
II. DATA ACQUISITION
The authors acquire the full Project Gutenberg collection automatically rather than relying on manual downloads. The pipeline also retrieves two kinds of book metadata and organizes each book at four levels of granularity.
II. DATA ACQUISITION
- The approach downloads the full PG collection automatically through a local mirror, using a single command.The paper says manual downloading does not scale reasonably to a collection of more than 50,000 books.Source: §II · p. 2
- The folder structure stores each book as raw data, text, tokens, and counts.The pipeline figure also shows example basic Python commands for preprocessing.Source: Figure 1 · p. 2
- The pipeline retrieves uploader-provided metadata on authors, language, subjects, and downloads, plus collaborative bookshelf tags.Bookshelves categorize books into collections such as “Art” or “Fantasy.”Source: §II · p. 2
III. DATA PROCESSING
As of July 18, 2018, processing yielded 55,905 books in four granularities. The stages remove duplicates and non-UTF-8 entries, strip boilerplate, tokenize the text, and count word occurrences.
III. DATA PROCESSING
- 55,905 books were processed as of July 18, 2018, in four levels of granularity.The levels are raw data, text, tokens, and counts.Source: §III · p. 2
- Raw-data processing removes duplicate entries and entries not encoded in UTF-8.Books are saved according to their PG-ID.Source: §III · p. 2
- Text processing removes headers and boilerplate text from the books.The paper refers readers to Methods for processing details.Source: §III · p. 2
- Token processing uses the NLTK tokenizer to produce a token time series without punctuation.The final count data lists each word type w with its occurrence count n_w.Source: §III · pp. 2–3
IV. DATA DESCRIPTION
The corpus spans long books, estimated publication dates, and many languages, while its topical labels differ in distribution and overlap. The authors find that bookshelf labels are more evenly distributed and more often unique to a book than subject labels.
IV. DATA DESCRIPTION
- 81% of books are tagged English; the corpus includes 56 languages, and the English collection contains 2.8 × 10^9 tokens.French is the second-largest language corpus, with more than 10^8 tokens.Source: §IV · p. 3, Figure 2 · p. 3
- A substantial fraction of PG books exceeds 10^5 word tokens, and most exceed 10^4.The length distribution has a heavy tail at large values.Source: §IV · p. 3, Figure 2 · p. 3
- Publication-year estimates place most books around 1900, with substantial numbers spanning 1800–2000.The estimate uses author birth and death years, rather than the book’s first-publication date.Source: §IV · p. 3, Figure 2 · p. 3
- Bookshelf labels are more evenly distributed across books than subject labels, which are usually assigned to fewer than 10 books per label.Bookshelf labels are assigned to 10–100 books in most cases.Source: §IV · p. 3, Figure 3 · p. 4
- Up to 85% of books have a unique bookshelf label, while roughly 50% have two or more subject labels.The authors suggest bookshelf labels are better suited to practical applications such as text classification.Source: §IV · p. 3, Figure 3 · p. 4
V. QUANTIFYING VARIABILITY IN THE CORPUS
The paper uses word-frequency distances to illustrate variability across bookshelf labels, authors, and time. The examples show topical and author clustering, while temporal patterns are qualified by publication-date and corpus-composition limitations.
Labels
- Books tagged with the same bookshelf tend to cluster in a UMAP embedding of pairwise word-frequency distances.The visualization covers 370 books across five bookshelf labels.Source: §V · p. 4, Figure 4 · p. 4
Authors
- For the 20 most prolific authors, same-author book pairs have consistently smaller distances and less spread than different-author pairs.The comparison draws 1,000 pairs of each kind per author.Source: §V · p. 4, Figure 5 · p. 5
Time
- Average distance increases with time separation, but the increase is substantial mainly for large gaps involving later periods after 1900.The authors note approximate publication years and a possible post-1930 change in corpus composition as qualifications.Source: §V · p. 4, Figure 6 · p. 5
VI. DISCUSSION
The discussion presents SPGC as a reproducible, multilingual resource built from Project Gutenberg and its metadata. It also identifies data boundaries that affect some uses, especially diachronic analysis.
Discussion
- SPGC combines more than 50,000 books in over 20 languages with metadata from two sources and provides code plus a frozen corpus version.The frozen SPGC-2018-07-18 version is intended to make the reported results reproducible.Source: §VI · p. 5
- The corpus contains relatively few post-1930 books, lacks exact publication dates, may include duplicate editions, and mixes genres.The authors note that bookshelf labels can help control genre composition, while missing exact dates hinder diachronic studies.Source: §VI · p. 5
- The authors position SPGC as a starting point for quantitative linguistics and note that some applications may require extra preprocessing such as stemming.They identify potential uses in computational linguistics, machine learning, and information retrieval.Source: §VI · pp. 5–6
Pre-processing
The preprocessing workflow downloads and standardizes Project Gutenberg texts, then removes boilerplate, tokenizes, filters, and lowercases tokens. Its modular filtering choices make the output customizable but impose specific token-level trade-offs.
Pre-processing
- The pipeline retrieves PG books and metadata, removes boilerplate, tokenizes texts, filters tokens, lowercases them, and counts word types.It keeps the latest version when an identifier appears in multiple server locations, but does not remove duplicates based on metadata or content.Source: Pre-processing · p. 6
- Tokenization uses NLTK's TreebankWordTokenizer by default and passes each book's language metadata to the tokenizer.For books marked as multilingual, the first language entry is used.Source: Pre-processing · p. 6
- Only alphabetic tokens, including accented characters, are retained and lowercased, excluding tokens with digits or other symbols.The authors note that this can discard years while avoiding page and chapter numbers; lowercasing can conflate proper nouns with common nouns.Source: Pre-processing · p. 6
- The filtering is modular and can be customized for application-specific needs, including adding stemming.The authors state that no single filtering solution fits every application.Source: Pre-processing · p. 6
2-dimensional embedding
The paper uses UMAP to project normalized word-count data into two dimensions, with books as points and word types as dimensions. Because bookshelf labels are not input to UMAP, observed clustering reflects patterns in word-frequency statistics.
2-dimensional embedding
- UMAP projects normalized word counts, with word types as features and books as samples, using Jensen-Shannon divergence for distances.Bookshelf labels are not passed to the embedding, so the observed clustering reflects word-frequency statistics rather than label input.Source: 2-dimensional embedding · p. 6
Data availability.
The authors provide a frozen SPGC snapshot to support reproducibility, alongside code for downloading and processing the growing Project Gutenberg collection. The full dataset, one-gram counts, and preprocessing run have distinct storage and compute requirements.
Data availability.
- The frozen SPGC snapshot contains 55,905 books, was last updated on 2018-07-18, and underlies all statistics and figures in the manuscript.The code can also download and process all available Project Gutenberg books, which are added daily, so local versions may differ by update date.Source: Data availability · p. 6
- The frozen dataset at all granularity levels occupies 65GB, while one-gram counts alone require 3.6GB.Source: Data availability · p. 7
- Preprocessing the frozen data took 8 hours without parallelization on a 3.40GHz CPU.Source: Data availability · p. 7