PaperlayerView exact revision on arXiv

Source-linked AI summary

Institutional Newspapers Pipeline: Deriving billions of high quality tokens from historical newspapers

Matteo Cargnelutti, Catherine Brobston, Eben English, Jake Sadow, Kacie Bailey, Greg Leppert, Amanda Watson, Jessica Chapel, Jonathan Zittrain

arXiv:2608.18972v1cs.CLcs.DL

TL;DR

Historical newspapers are difficult to extract computationally at scale because their layouts are dense, irregular, and noisy. The paper presents a modular pipeline that processes scans into structured crops and releases 16.3 billion OCR tokens from 1,473,635 scans as an open dataset.

  • Problem

    Dense, irregular, and variable-quality newspaper layouts make large-scale computational extraction difficult for general-purpose tools.

  • Method

    The pipeline segments scans into atomic crops and applies OCR, text and image classification, reading-order detection, entity recognition, subject classification, language detection, and embeddings generation.

  • Results

    16.3 billion o200k_base OCR tokens were produced across 83.1 million crops from 1,473,635 historical newspaper scans.

  • Takeaways & Limitations

    The work represents a meaningful step toward unlocking high-quality data from abundant but underused historical newspapers.

  • Takeaways & Limitations

    Users must independently assess legal restrictions, including copyright, trademark, privacy, and publicity rights, when using the dataset.

Abstract

from arXiv · show

Historical newspapers are an abundant record of public life, but their dense, irregular and sometimes noisy layouts make computational access to these materials both challenging and limited. We present the Institutional Newspapers Pipeline, a modular system we jointly designed with Boston Public Library to extract high-quality, structured datasets from historical newspaper scans. It was architected so that each step remains interpretable and customizable, and so that the pipeline as a whole remains computationally frugal enough to run on workstation-level hardware. The pipeline runs each scan through a multi-step process: it segments scans into individual type-agnostic crops and performs OCR on each resulting segment before then performing text analysis, type classification, reading order detection, named entities recognition, subject classification, language detection, and pre-computed embeddings generation on every crop. We ran this pipeline against a portion of Boston Public Library's holdings and released the results as an open dataset. The optical character recognition (OCR) output represents 16.3 billion o200k_base tokens across 83.1 million individual crops, extracted from 1,473,635 public domain newspaper scans published between 1795 and 1930. This report describes our methods for each processing step, the small models we trained, as well as the evaluation results and dataset-scale measurements we collected in the process. It accompanies the release of the pipeline, models, and dataset. We position this work as a substantial step towards unlocking high-quality data from tens of millions of newspaper scans.

1 Introduction · 2 Contributions · 3 Segmentation

The report presents a modular, interpretable, and computationally frugal pipeline for converting difficult historical newspaper scans into structured crop-level data. It introduces the pipeline and dataset, then describes segmentation methodology and collection-scale results.

  • 1 Introduction: Historical newspapers are difficult to process computationally because scans vary in quality and use dense layouts, columns, decorative headings, and tables.Extracted text is often too low quality for more than unreliable keyword search, limiting research into recorded historical events.
  • 1 Introduction: The report positions high-quality newspaper data as useful for improving AI’s “digital diet” and enabling use cases beyond model training, including patron access and research.The motivation connects both the scale of language-model data needs and the influence of pre-training data quality on model performance and behavior.
  • 1 Introduction: The pipeline was jointly designed by the Institutional Data Initiative and Boston Public Library to improve patron and researcher access through an “atomic” understanding of scan contents.The system defines an atomic item as a crop and processes scans through segmentation, OCR, text analysis, classification, reading-order detection, entity recognition, subject classification, language detection, and embeddings generation.
  • 2 Contributions: The release contributes a crop-level dataset containing bounding boxes, crop types, OCR outputs, reading-order indexes, named entities, subject classes, and text and language metrics.The dataset was generated from Boston Public Library’s public-domain newspaper collection, with accompanying pipeline methods and trained models described in the report.
  • 3.1 Methodology: The segmentation goal is to detect rectangular crops around uninterrupted text or visual flows, keeping anchored headlines with articles and visual-heavy advertisements generally self-contained.Segmentation is separated from classification to better leverage limited annotations and avoid class imbalance during full-scan training.
  • 3.1 Methodology: 1,020 scans were annotated with 47,485 bounding boxes, and the final model used 15% of scans for validation and another 15% for evaluation.The model used YOLO26x with 55.7M parameters, trained at 960 pixels after testing resolutions from 320px to 1280px.
  • 3.1 Methodology: 153 held-out scans containing 2,991 bounding-box instances were used for evaluation, and the report states that the model effectively segments this collection.Eigen-CAM analysis showed strongest activation on mastheads and headline banners, with secondary activation following vertical column structure and a dense advertisement block.
  • 3.2 Results: 83,147,041 crops were identified across 1,473,635 scans, averaging 56.4 crops per scan, while crop detections covered an average 89% of each scan.Mid-19th-century scans appear densest by average crop count, and uncovered outer borders measured 2.6% to 3.6% on each side in a sample.

4 Crop-level OCR

The pipeline uses crop-level OCR with both Tesseract and a specialized vision-language model to address newspaper layouts that challenge traditional OCR. dots.mocr produced more OCR tokens than Tesseract across the collection, consistent with recovering text from dense, decorative, and damaged crops.

  • Motivation and approach: Traditional OCR often failed to detect columns and parse large decorative text, motivating a crop-level approach using both Tesseract and a specialized VLM.Full-scan VLM OCR showed promising out-of-the-box results, but the pipeline focused on crop-level OCR.
  • Tesseract OCR: Tesseract 5 with tessdata_best models was selected because its word-level bounding boxes support OCR outputs compatible with library discovery systems such as AltoXML.Unlike VLMs, Tesseract rarely hallucinates at the level of a full word or sentence.
  • VLM OCR: dots.mocr was selected after sample review because it is a 3B-parameter multilingual VLM that can parse tables.The team replaced dots.ocr with dots.mocr shortly after dots.mocr was released; dots.mocr can still hallucinate and enter token loops.
  • Implementation: Tesseract processing used multiprocessing through tesserocr, with crop images downscaled to a maximum of 1.5 megapixels to balance accuracy and computational footprint.dots.mocr inference used vLLM across 8 NVIDIA L40S GPUs, with image sizes clamped between 0.25 and 1 megapixel.
  • Collection-scale output: 16.3 billion tokens were produced by dots.mocr and 14.7 billion by Tesseract using the o200k_base tokenizer.With few exceptions, dots.mocr produced more tokens both in total and per scan, consistent with recovering text from dense crops, decorative elements, and damaged scans.

5 Crop-level Language Detection and Text Analysis

The pipeline detects languages at crop level and computes standardized text metrics for comparing Tesseract with dots.mocr. Language assignment covered nearly all crops, while dots.mocr generally recovered more text and emitted structured Markdown output.

  • Text Analysis: Crop-level analysis measured character counts, total and unique word and sentence counts, tokenizability, and dots.mocr Markdown or table markers.Metrics were collected for both OCR engines, except Markdown and table detection, which applied only to dots.mocr output.
  • Text Analysis: Flattening removed markup, rejoined line-end hyphenations, replaced line breaks with spaces, removed zero-width spaces, and used detected language for PyICU splitting.English was used as a fallback when no language was available.
  • Language Detection: 99.81% of crops received language codes, with Lingua directly detecting 96.96% and issue-level metadata supplying the remaining 3.04%.English accounted for 97.61% of crops with a language code.
  • Language Detection: 73 distinct language codes revealed a multilingual tail led by Yiddish, German, Swedish, and French.The ten most common languages represented 99.97% of coded crops, leaving 0.03% across the other 63 codes.
  • OCR Comparison: Compared on the same crops, dots.mocr recovered more characters, words, and sentences and had slightly higher mean tokenizability, while Tesseract had more unique words.Dots.mocr output contained Markdown tables in 0.78% of crops and other Markdown markup in 3.33%; these figures also indicated missed formatting for some tables and headings.

6 Crop-type Classification

Crop-type classification combines image and OCR-text signals through two small classifiers, improving interpretability and supporting category-sensitive decisions across 83,147,041 crops. The resulting classifications show strong agreement, complementary modality strengths, and distinct differences between crop frequency, token contribution, and surface area.

  • Models and training: Two small classifiers use complementary signals: YOLO26m-cls image features and a Model2Vec-fine-tuned potion-base-32M text model.Both models target broad categories such as Content, Advertisement, and Photograph or illustration to limit class-imbalance effects.
  • Models and training: 188,477 image annotations and 185,900 text annotations trained the classifiers from auto-annotated crops, using targeted 70/15/15 splits.A final manual check of 600 annotations achieved 91% accuracy under strict checks.
  • Evaluation: 0.84 versus 0.48 F1 for Photograph or illustration and 0.92 versus 0.68 for Cartoon show the image classifier’s advantage on visual categories.The classifiers agree closely on text-heavy categories including Advertisement, Content, and Section heading.
  • Combined decisions: 90.55% agreement across 83,147,041 crops supports combining both classifiers, while the final decision follows the image classifier on 95.03% and the text classifier on 95.51% of crops.Weighted average confidence is 0.953 for the image classifier and 0.967 for the text classifier.
  • Dataset-scale patterns: Advertisements are the most common crop type, but Content crops contain most OCR tokens in nearly all decades, despite advertisements outnumbering them in most decades.From the 1870s onward, advertisements occupy more surface area than their token share, reaching a gap of about 15 percentage points by the 1920s.

7 Scan-level Reading Order Detection

The pipeline uses a fast HDBSCAN-based, content-aware method to infer scan-level reading order despite highly variable newspaper layouts and disconnected crop relationships. It achieves strong sequencing agreement with annotations while remaining computationally limited, but struggles with deeply complex or unique layouts.

  • Challenges: Reading order is challenging because newspaper layouts vary widely and only some crops on a page are directly connected.The pipeline operates on individual crops, including article fragments spread across columns or pages.
  • Method: The method combines HDBSCAN clustering with content-aware post-processing to produce a fast reading-order sequence.It classifies crops by width, clusters narrow content crops into columns, assigns other crops to columns, orders columns and crops, then nudges positions using left-edge bucketing.
  • Evaluation: 72.1% macro and 80.8% micro position accuracy were achieved, with Kendall's τ of 0.922 and 0.959, respectively.Macro scores average scans equally, whereas micro scores pool all crops and therefore weight scans by crop count.
  • Evaluation: Kendall's τ indicates that overall sequencing can match annotations even when exact crop positions differ.Position accuracy requires each crop to occupy exactly the correct sequence index, while Kendall's τ assesses broader ordering agreement.
  • Limitations: The method has a limited computational footprint but cannot properly handle deeply complex or unique layouts.A transformer-based approach is proposed as a future direction for these cases.

8 Crop-level Named Entity Recognition

The pipeline adds experimental crop-level named entity recognition to support granular analysis of newspaper scans, using an off-the-shelf model with confidence filtering, entity-type restrictions, and deduplication. It detected hundreds of millions of location, person, and organization mentions, but the results remain subject to historical-domain, OCR, and harmful-language limitations.

  • Motivation: NER was added to provide an “atomic” understanding of newspaper scans and enable granular downstream use cases.The motivation was aligned with Boston Public Library’s goals for the collection.
  • Method: Flair’s base ner-fast model processed VLM-extracted text from each crop, making the output experimental because collection-specific NER development was out of scope.Detections below 0.85 confidence were discarded; only PER, LOC, and ORG entities were retained, with likely duplicates reduced to the highest-confidence detection.
  • Results: 155.6 million location mentions, 142.2 million person mentions, and 49.1 million organization mentions were detected across the collection.Locations had the fewest unique surface forms relative to their volume, consistent with recurring place names.
  • Results: The most frequent entities were coherent with the collection, including Boston-area and major national locations, government organizations, and common period family names.Examples included “Boston”, “Mass”, “New York”, “Washington”, “United States”, “Congress”, “Senate”, and “House”.
  • Limitations: Historical out-of-distribution names, OCR errors, and occasional harmful-language detections limit reliability, so the NER data should assist downstream research but not be taken at face value.The paper explicitly characterizes the technique and resulting data as experimental.

9 Crop-level Subject Detection

The pipeline assigns experimental, configurable subject labels to newspaper crops using zero-shot classification of VLM-extracted text. Across predicted crops, business, commercial, scientific, and political subjects dominate, while confidence varies widely.

  • Method: Subject detection is experimental and designed to be rerun with a collection- or research-specific label list.The selected subjects reflect observations from this collection and may need adjustment for specific research needs.
  • Method: The pipeline applies ModernBERT-large zero-shot classification to each crop’s VLM-extracted text and stores full label rankings with confidence scores.Classification uses Hugging Face’s zero-shot pipeline and runs when VLM-extracted text is available.
  • Results: Business, commercial, scientific, and political labels are the most frequent top-ranked subjects across crops.Commercial advertisements and mastheads overlap by design with crop-type categories, providing an independent signal and fallback for ambiguous crops.
  • Results: 0.65 is the mean top-1 confidence across 82,985,966 predicted crops, while per-label standard deviations range from 0.08 to 0.23.The wide spread is expected in zero-shot classification because scores derive from textual entailment rather than a task-specific trained distribution, motivating the experimental designation.

10 Crop-level Embeddings · 11 Crop-level Detection of Chronicling America's Thesauri Terms

The pipeline pre-computes generic image and text embeddings for every crop to support dataset-wide vector search and downstream use, while also detecting thesauri terms to aid digital humanities research. Embedding storage scales to roughly 213 GB, and thesauri detections are explicitly experimental navigational aids rather than textual interpretations.

  • 10.1 Methodology: Pre-computed image and text embeddings were generated for each crop to enable dataset-wide vector search and facilitate downstream use.The embeddings were produced for both the visual and textual content of each crop.
  • 10.1 Methodology: Text embeddings use minishlab/potion-multilingual-128M through Model2Vec, a generic static model with a 128M-parameter footprint.The model was selected for its light footprint and overall genericity.
  • 10.2 Results: 384-dimensional image and 256-dimensional text vectors require roughly 2,560 bytes per crop, totaling approximately 128 GB and 85 GB respectively across 83.1 million crops.The combined storage requirement is roughly 213 GB, with image embeddings occupying about 1.54 GB per million crops.
  • 10.2 Results: The static text model lacks attention and relies heavily on token distribution, reducing usability in semantically ambiguous contexts.Both embedding models are generic, and specialized applications may benefit from domain- or task-specific models.
  • 11 Crop-level Detection of Chronicling America's Thesauri Terms: The pipeline detects race, ethnicity, immigration, and citizenship thesauri terms against VLM-extracted text from each crop.The thesauri were developed with Harvard Law School Library’s Public Data Project from collections originally compiled by Library of Congress, National Endowment for the Humanities, and state partners.
  • 11 Crop-level Detection of Chronicling America's Thesauri Terms: These thesauri matches support digital humanities research as navigational and statistical aids for contextualizing historical content and language.The detections are experimental and “naive” and should not be used as-is to infer the nature of a text.

12 Pipeline Footprint · 13 Dataset Preparation and Post-Processing · 14 Discussion and Future Directions

The pipeline was computationally substantial but designed for workstation-grade hardware, with OCR dominating runtime and an estimated rented-hardware cost of approximately $25,000. Dataset post-processing improved usability, while the authors identify collection generalization and VLM-based OCR efficiency as priorities for future work.

  • 12 Pipeline Footprint: OCR dominated processing time: dots.mocr took 28.62 minutes and Tesseract 16.48 minutes per 200-issue batch, together about half of the 86.69-minute full batch.The remaining thirteen steps were comparatively inexpensive, most running in a few minutes or less.
  • 12 Pipeline Footprint: Approximately $25,000 would cover the collection’s estimated ~1650-hour runtime on rented 8xL40S GPU hardware.The estimate assumes an average rental cost of ~$15/hour.
  • 13 Dataset Preparation and Post-Processing: Post-processing included VLM OCR cleanup, locality-metadata correction, and confidence- and word-count-based language-code filtering using Library of Congress metadata.OCR cleanup addressed dehyphenation, Markdown heading normalization, and token loops; locality corrections included erroneous city-state associations.
  • 13 Dataset Preparation and Post-Processing: 2,521,901 crops, or 3.04% of crops carrying a language code, received replacement language codes under the post-processing rules.Replacement codes were stored without confidence scores to distinguish them from direct detections.
  • 14 Discussion and Future Directions: The authors position the work as a meaningful step toward unlocking high-quality data from abundant but underused historical newspapers.They frame the pipeline, models, and data as resources for broader community use.
  • 14 Discussion and Future Directions: Because the models were developed and tested on a Boston Public Library subset, the pipeline may struggle with materially different newspaper scans.The authors plan to revise methods and repeat the experiment on other collections with library partners.
  • 14 Discussion and Future Directions: Future work targets computational-footprint reduction, especially VLM OCR, through a smaller, more accurate, and less token-loop-sensitive model specialized for historical newspaper crops.The authors hypothesize that task-specific VLM training could improve these properties over dots.mocr.

Rights determination · Disclaimers · Appendices

The dataset was assembled from materials assessed as public domain in the United States, but users must independently evaluate legal, ethical, and data-quality risks. Disclaimers address jurisdictional rights, harmful historical content and metadata, collection bias, and generated or experimental outputs.

  • Rights determination: The dataset includes issues published before 1931 and assessed as public domain in the United States using Boston Public Library issue-level metadata.The assessment was made in good faith from available information.
  • Rights determination: Some materials may remain protected by copyright or other rights in jurisdictions outside the United States, and missing rights claims do not establish public-domain status.Copyright-status information reflects available data at the time of determination.
  • Disclaimers: Users are solely responsible for independent legal assessments, permissions, and compliance with trademark, privacy, publicity, and other applicable restrictions.Rights holders may report possible misidentification or infringements in the release.
  • Disclaimers: Historical materials may contain outdated, offensive, harmful, discriminatory, or inaccurate language and portrayals that reflect their original contexts.Examples include racism, sexism, and colonial attitudes.
  • Disclaimers: Collection metadata may reproduce harmful, outdated, biased, or incomplete language because descriptions and terminology derive from historical sources and cataloging practices.The paper notes that collecting, describing, and classifying library materials are shaped by personal, institutional, and societal biases.
  • Disclaimers: The dataset contains generated or experimental content that may include errors or inaccuracies and is provided as is without warranties.Users should independently verify the data and apply their own judgment.

Appendix A — Outer scan margins … Appendix D — Compute environment

The appendices quantify scan-margin coverage, document crop-type annotation and confidence behavior, and specify the hardware used to run the pipeline. They show that uncovered area is concentrated in outer margins, classifier uncertainty is category-dependent, and processing used a single GPU node.

  • Appendix A — Outer scan margins: 100 randomly sampled scans were measured to estimate uncovered outer margins, with no scan discarded for missing dimensions or absent crops.The sample drew one scan from each of 100 randomly selected metadata shards using seed 0.
  • Appendix A — Outer scan margins: 88.02% of each scan’s area was enclosed by all crop detections on average, ranging from 75.84% to 95.92%.This envelope was slightly smaller than the 89% average crop coverage reported in Section 3.2.
  • Appendix A — Outer scan margins: The results indicate that most uncovered area lies in outer margins, leaving comparatively little content uncovered within the printed area.Crops nearly fill the envelope enclosing them.
  • Appendix B — Crop-type auto-annotation prompt: Qwen/Qwen3-VL-30B-A3B-Thinking-FP8 was prompted to auto-annotate the crop-type classification training set using each crop’s OCR text.The OCR text was appended after the prompt’s Text: marker.
  • Appendix C — Crop-type classifier confidence: Confidence was highest for the high-volume Advertisement and Content categories and the structural Section heading and Masthead, nameplate or running head categories.Confidence dropped sharply for Photograph or illustration and Cartoon, which also had the weakest precision and recall in Table 4.
  • Appendix C — Crop-type classifier confidence: 1.000 ± 0.000 was an artifact for the text classifier’s Empty category, because Empty was assigned by rule rather than learned.The image classifier learned Empty, whose 0.679 mean was lowest among the seven categories and consistent with its low precision.
  • Appendix D — Compute environment: The pipeline ran on a single Institutional Data Initiative GPU node equipped with 8 × NVIDIA L40S GPUs, 256 CPU cores, and 768 GB of RAM.These specifications define the compute environment used for the pipeline run.

Appendix E — Dataset field list

The dataset is organized one row per newspaper scan, with crop-level fields stored in reading-order-aligned lists and issue-level metadata, OCR, and text-analysis fields documenting each page and crop.

  • Dataset organization: Each dataset row represents one newspaper scan, while most crop-level fields are lists aligned to crop_bbox_gen’s reading order.NER, classification, subject, and language fields use lists of lists, with one inner list per crop.
  • Issue metadata: Issue-level fields record publication date, edition, metadata source, locality, and issue-level language.The listed fields include year, month, day, edition, metadata_source_gen, city_gen, state_gen, country_gen, and language_ext.
  • Crop fields: Crop geometry and detection quality are represented by crop_bbox_gen bounding boxes and crop_bbox_conf_gen detection confidence scores.Both fields are crop-level lists associated with the detected segments.
  • OCR fields: The dataset provides post-processed dots.mocr OCR and Tesseract OCR, including per-word text, confidence, and bounding-box metadata for Tesseract.Tesseract word boxes use crop-relative pixel coordinates rescaled to full crop resolution.
  • Text analysis: Per-crop text analysis includes tokenizability, character, word, lexical-ratio, and sentence metrics, plus table and Markdown indicators for both OCR sources.Metrics are prefixed tesseract_ and vlm_ where applicable, with vlm_has_table and vlm_has_markdown included.
Loading 2608.18972v1…