PaperlayerView exact revision on arXiv

Source-linked AI summary

Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale

David Lowry-Duda, Matteo Cargnelutti, Catherine Brobston, Salwa Ismail, Greg Leppert, Amanda Watson, Jonathan Zittrain

arXiv:2608.19026v1cs.CLcs.DL

TL;DR

Existing web-oriented preprocessing can remove multilingual content, paratext, and document structure from historical book collections. This paper introduces Enriched Text, an annotated and configurable alternative, and shows that it supports research on textual reuse and historical language change.

  • Problem

    Web-oriented preprocessing often aggressively filters languages, deduplicates text, and discards paratext, limiting fidelity for multilingual historical book collections.

  • Method

    Enriched Text preserves OCR text, paratext, metadata, detected languages, and duplicate information through machine-readable annotations that users can configure.

  • Results

    The enriched dataset reveals cross-lingual textual reuse and supports analysis of historical language change using duplicate-paragraph annotations and per-paragraph BPB.

  • Takeaways & Limitations

    Annotating rather than removing information makes the dataset adaptable to different research needs and opens analyses unavailable from a flattened token stream.

  • Takeaways & Limitations

    Users are responsible for independently assessing legal rights and obtaining permissions where required for their uses of the materials.

Abstract

from arXiv · show

Released in 2025, Institutional Books: Harvard Library (IB-HL) is a collection of 983,004 volumes (242B o200k_base tokens), originally digitized through Harvard Library's participation in the Google Books Library project. As researchers and developers have begun to use IB-HL, a tension has emerged between standard large-scale preprocessing practices and the goals of careful information stewardship. Many existing pipelines optimize for web text: as a result, they tend to aggressively filter, deduplicate, restrict by language, and sometimes discard meaningful metadata. Meanwhile, researchers seeking to use IB-HL duplicate effort while performing similar processing and analysis. We describe an approach that we call Enriched Text. Instead of producing a single 'complete' stream of tokens, we normalize the text while preserving metadata through annotations. We separate endmatter, detect per-paragraph language, identify clusters of duplicate paragraphs, and compute per-paragraph bits-per-byte scores. We provide this information through HTML-like annotations layered on top of the text. By parsing these annotations, users can tailor the output to their own needs instead of accepting a global editorial decision on content. The pipeline applies to all $\approx$250 languages in the collection. This report describes this project's goals, implementation, and design rationale. The release includes IB-HL-ET (an enriched-text version of IB-HL containing 217B o200k_base tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs) and the pipeline that produced it. These serve to make the collection easier for machines to parse and for humans to study.

1 Introduction

The paper introduces Enriched Text as a configurable alternative to globally “cleaning” IB-HL, preserving OCR text structure, metadata, and duplicate information through machine-readable annotations. IB-HL-ET addresses multilingual OCR cleaning, semantic segmentation, and collection-wide duplicate identification for computational access and scholarship.

  • Motivation: 242B-token dataset of digitized books and bound materials, IB-HL emphasizes provenance, collection-level deduplication, OCR artifact analysis, legibility improvements, and rich metadata.The collection was introduced by the Institutional Data Initiative in 2025.
  • Problem: Standard internet-oriented pipelines can aggressively filter languages and discard near-duplicate text, creating tension with IB-HL’s information-stewardship goals.Early-adopter feedback motivated an alternative preprocessing approach.
  • Approach: Enriched Text offers a configurable dataset by layering machine-readable annotations for paratext, metadata, detected languages, and duplicate paragraphs onto OCRed text.The approach avoids producing a single “clean” token stream and attempts to separate endmatter from main text.
  • Foundational problems: The pipeline targets multilingual OCR cleaning, semantically meaningful segmentation that respects document boundaries, and duplicate-segment identification across the collection.These are presented as three foundational problems in large-scale dataset construction.
  • Release: IB-HL-ET extends IDI’s community-led, knowledge-institution-anchored data project into information-access workflows, training pipelines, computational scholarship, and related forms of access.The release is presented as a variant of IB-HL designed for modern computational use.

2 Contributions

This section presents a technical report, the IB-HL-ET enriched-text dataset, and the Python pipeline created to produce it. The dataset spans 217B o200k_base tokens across 7B sentences, 1.39B subtopic paragraphs, 297M subtopic sections, and 983k volumes.

  • Contributions: The project contributes a technical report documenting the processing work used to create IB-HL-ET, an Enriched Text version of the Institutional Books — Harvard Library dataset.The dataset is sourced from Harvard Library’s collection.
  • Contributions: 217B o200k_base tokens span the public IB-HL-ET dataset’s 7B sentences, 1.39B subtopic paragraphs, 297M subtopic sections, and 983k volumes.The dataset is publicly available through Hugging Face.
  • Contributions: The project contributes the Python pipeline created to produce IB-HL-ET.The supplied passage identifies the pipeline as a project contribution but truncates its further description.

3 Goals and Scope

IB-HL-ET aims to make the Harvard Library collection easier to access computationally by organizing existing OCR text without re-OCRing volumes. Its design prioritizes configurable annotations and source fidelity while favoring widely available, frugal computation.

  • Goals and Scope: The pipeline organizes and analyzes existing OCR text to make the collection easier to read, filter, search, and study computationally, including with AI.It treats the work as text-to-text processing and does not re-OCR volumes.
  • Configurability: Annotations separate linguistic content from paratext, identifying endmatter and middlematter so users can retain context while choosing how to process it.The project labels material rather than deleting it, supporting different computational requirements.
  • Fidelity and Tradeoffs: The project prioritizes fidelity by marking and annotating material, but removes page numbers and running headers and footers because they become recurring noise after page boundaries are removed.Unmodified OCR remains available in IB-HL for users whose requirements differ.
  • Available Compute: One month is the approximate target for reproducing the whole collection on a single NVIDIA DGX Spark, using mostly CPUs or small single-GPU models.Computing bits-per-byte for all paragraphs is the major exception: even with a 0.6B model, it requires months of dedicated GPU time.

4 Processing Pipeline

The processing pipeline applies sequential text-to-text transformations to IB-HL, combining multilingual modeling with normalization, deduplication, classification, dehyphenation, and header/footer removal. It preserves broad language coverage while documenting substantial computational costs, dataset limitations, and stage-specific error patterns.

  • Pipeline overview: The pipeline processes 242B o200k_base tokens across 983,004 volumes through a sequence of text-to-text transformations.The section describes the aims and methodology of each processing step, with additional details in Appendix C.
  • Multilingual modeling: 388 base models and two endmatter classification models were trained, covering n-gram modeling in 250 languages and Nupunkt segmentation in 138 languages.Base models used subcollections of up to 30 books per language; one endmatter classifier and one subclassifier were also trained.
  • Endmatter classification: 0.97 accuracy (F1 0.97) was achieved by the endmatter classifier, compared with 0.90 overall accuracy for the subclassifier.Endmatter classification lasted 211h, and errors more often moved endmatter into middlematter than the reverse.
  • Normalization and deduplication: 46h was spent on soft normalization, while page-level deduplication required 164h across transient hard normalization, simhash computation, and pairwise simhash comparison.Hard normalization is lossy, used only internally for matching passages, and never stored or presented to users.
  • Dehyphenation: 2,322,246,287 total hyphens were removed, with an estimated EOL hyphen removal rate of ≈89%.Dehyphenation required 640h; English books averaged 4.9 hyphens merged per page, while Chinese, Japanese, and Arabic averaged below 0.1 removed hyphens per page.
  • Header and footer removal: 272,583,638 total header and footer lines were removed from 760,238 volumes (77.3%), averaging 277 removed lines per volume.Running header and footer removal required 927h at an average of 3.4s/volume, with lower detection accuracy in some vertically oriented languages and language-specific OCR placements.

5 Dataset Analysis

The dataset analysis validates the pipeline through corpus statistics, cross-lingual duplicate reuse, and BPB trends across publication history. IB-HL-ET improves tokenizability, reveals mostly same-language reuse, and shows declining text predictability toward the present while retaining important confounds.

  • Overview: Pipeline analyses compare IB-HL-ET with IB-HL, measure duplicate reuse across languages, and relate per-paragraph BPB to publication metadata.Each analysis depends on a different annotation layer added in the pipeline.
  • Text statistics: ≈26B fewer characters distinguish IB-HL-ET from IB-HL, primarily because the pipeline separates endmatter and removes headers, footers, and page numbers.The comparison uses final middlematter with annotations removed.
  • Text statistics: +9.5, +5.8, and +5.6 are the largest tokenizability increases for Occitan, Persian, and Arabic, respectively.Tokenizability increased for every language; differences in bigram and trigram uniqueness were small.
  • Duplicate reuse: 5.3% of 124,049,981 duplicated paragraphs have detected languages differing from their host volumes’ primary languages.Among paragraphs at least 200 characters, the corresponding cross-language share is 2.3%; short fragments can produce detection artifacts.
  • BPB trends: Median per-volume BPB falls from ≈2.5 in the 17th century to ≈1.4 in the mid-19th century and ≈1.0 by the mid-20th century.English follows approximately the same shape with a slightly smaller median, while OCR noise and post-1930 sample selectivity confound interpretation.

6 Discussion and Further Directions

The discussion frames IB-HL-ET as an extensible, collaborative starting point that preserves paratext and metadata through annotation rather than removal. It highlights research opportunities from enriched annotations, current pipeline limitations, and collaborations aimed at broader computational access and improved low-resource-language models.

  • Discussion and Further Directions: IB-HL-ET is presented as an ongoing collaborative process and a customizable starting point rather than a finished artifact.Its annotation-over-removal philosophy invites additional annotations and analyses as new use cases arise.
  • Analysis of the Enriched Text: Marked duplicate paragraphs make the collection a map of textual reuse, revealing rich cross-lingual quotation and clustering structures.The enriched dataset also supports analyses unavailable from a flattened token stream, including pairing per-paragraph BPB with publication year.
  • Pipeline Improvements: Current weaknesses include recognizing tables, marginalia, and technical language.The report points to two known examples in Appendix D and expects advances in OCR to improve the pipeline.
  • Pipeline Improvements: ≈1.64 was the average BPB among the 21 other English volumes from the 1660s, contrasting with the decade’s unusually low overall BPB.The low value reflects 58 Acts and Resolves volumes with high OCR quality and formulaic statutory prose.
  • Pipeline Improvements: English OCR is relatively consistent, so the English BPB curve suggests a clear trend rather than merely OCR noise.The passage states that disentangling the observed variations requires additional research.
  • Towards a Virtuous Cycle: The project seeks diverse collaborations to make millions more books computationally accessible and improve analysis across the collection through better low-resource-language training data.The authors hope improved training data will lead to improved models and more accurate analysis.

8 Rights Determination

The dataset includes volumes identified as having no known copyright restriction based on specified HathiTrust statuses, but rights determinations may be complex, context-dependent, and occasionally erroneous. Users are responsible for independent legal assessments, permissions, and compliance with rights beyond copyright.

  • Selection basis: The release deliberately includes volumes identified by HathiTrust as “public domain,” “public domain in the United States,” or “CC-Zero.”These statuses were used to identify volumes for which there is no known copyright restriction.
  • Determination limits: Copyright determinations are complex, context-dependent, and occasionally subject to error.Rights information reflects available data at the time of determination and is provided on a good-faith basis.
  • Jurisdictional limits: Some volumes may be public domain in the United States while remaining protected by copyright or other rights in other jurisdictions.The absence of an explicit copyright claim or rights status does not guarantee public-domain status in the United States or abroad.
  • User responsibilities: Users bear sole responsibility for independent legal assessments, determining where and how to use materials, and securing any needed permissions.Potential restrictions may involve trademark, privacy, publicity, or other rights in addition to copyright.

9 Disclaimers

The dataset contains historical materials and metadata that may reflect outdated, offensive, harmful, or biased language and perspectives. It also includes generated or experimental content that may contain errors, so users should verify it independently and exercise their own judgment.

  • Historical content: Historical works may contain outdated, offensive, harmful, or inaccurate language, portrayals, and information, including racism, sexism, and colonial attitudes.These materials reflect the language, imagery, culture, and perspectives of their time.
  • Metadata: Collection metadata may use harmful, outdated, biased, or incomplete language, including terminology that differs from currently preferred community language.Descriptions may reuse language supplied by creators, including wording created decades ago.
  • Collection practices: Collecting, describing, and classifying library materials can introduce personal, institutional, and societal biases, including offensive subject headings, titles, and descriptions.Harmful viewpoints may also appear in the books themselves.
  • Data quality: Generated or experimental content is provided as is without warranties and may contain errors or inaccuracies, so users should verify the data independently and apply their own judgment.The creators state that reasonable care was taken to ensure quality.

Appendices … C.1 Preprocessing Details

The appendices provide technical details and design rationale, including dataset-field conventions, related-work context, and preprocessing implementation choices. The preprocessing uses language reporting for stratification and separates CPU- and GPU-oriented processing paths.

  • A Dataset Fields: Dataset fields distinguish source inputs from pipeline-generated values, while text statistics are computed over middlematter.The suffixes _src and _gen mark the two field origins.
  • B Additional related work: Related work spans boilerplate detection, OCR post-analysis, statistical methods, and newer machine-learning techniques for dataset cleaning.The cited literature includes computationally inexpensive boilerplate detection and OCR detection research.
  • B Additional related work: Large-scale data curation commonly involves sourcing, deduplication, cleaning, and filtering, with C4 adding heuristic cleaning, English restriction, and chunk-based deduplication.
  • B Additional related work: Limited transparency about training data motivates benchmarks such as DCLM, which supports comparison of data-curation practices.
  • B Additional related work: The approach was shaped by research progressing from large-data curation to filtering, extraction, and post-creation analysis.
  • Appendices: The appendices provide additional technical details and design rationale for the processing pipeline.
  • C.1 Preprocessing Details: Language reporting from IB-HL is used for stratification, and each base model is updated for every processed book.Small amounts of noise in base models have minimal effect on resulting pipeline behavior.
  • C.1 Preprocessing Details: Shards of 200 books are split into Nupunkt-compatible and non-Nupunkt-compatible languages because the former run on CPUs while the latter require GPUs for timely sentence segmentation.After this sharding, subsequent processing except deduplication is completely independent.

C.2 Unicode Normalization … C.7 Page Number Removal

The pipeline combines conservative and hard Unicode normalization, duplicate-page detection, endmatter and dehyphenation processing, running-header/footer matching, and cross-script page-number removal. Its design balances machine-readable normalization and deduplication with preservation of linguistically meaningful characters and explicit handling of multilingual edge cases.

  • C.2 Unicode Normalization: C.2 Unicode Normalization: Soft normalization applies NFC while preserving line breaks, curly quotes, hyphens, dashes, ligatures, and accents.It also removes selected zero-width characters, condenses Unicode spaces, collapses consecutive spaces, and strips per-line boundary spaces.
  • C.2 Unicode Normalization: C.2 Unicode Normalization: Hard normalization applies NFKC and converts punctuation, whitespace, line endings, tabs, and newlines toward a single ASCII-punctuated line.Unlike soft normalization, it removes soft hyphens and folds curly quotes, dash-like characters, and other Unicode variants.
  • C.3 Duplicate Page Removal: C.3 Duplicate Page Removal: Pages need at least 50 non-whitespace Unicode code points after hard normalization before duplicate checking.This lower bound prevents excessive removal of short content but mildly biases detection against languages carrying more meaning per character.
  • C.3 Duplicate Page Removal: C.3 Duplicate Page Removal: Character 9-gram simhashes classify pages differing by 6 or fewer bits as duplicates.The 9-gram choice balances sensitivity to exact content against random short-phrase collisions, while the strict threshold is intended to make accidental matches rare.
  • C.4 Endmatter Separation: C.4 Endmatter Separation: Batches of approximately 1024 pages are optimal at 7200 pages/s per CPU core, while typical book batches near 350 reach 6800 pages/s per CPU core.Memory bandwidth limits scaling, so using 20 cores is not 20 times faster than using 1 core.
  • C.5 Dehyphenation: C.5 Dehyphenation: Combining per-book and per-language base statistics improves dehyphenation decisions because per-book models lack reliable statistics for long words appearing only once.The model uses a naive joint probability estimator with add-k smoothing and fixed Stupid Backoff rather than a proper conditional-probability model.
  • C.5 Dehyphenation: C.5 Dehyphenation: A full Kneser-Ney KenLM prototype was substantially more expensive, adding approximately 10s per book, while the simpler estimator nearly always produced the same decisions.This tradeoff motivated using the computationally cheaper estimator.
  • C.6 Running Headers and Footers: C.6 Running Headers and Footers: A Jaccard similarity threshold of 0.85 allows approximately one typo or one digit change in page numbers and was conservatively effective in experiments.The implementation uses datasketch’s default SHA-1 hashing; LSH construction, especially NumPy permutation-and-min operations, is the bottleneck.

C.8 Segmentation Details · C.9 Alternate Chunking Algorithms Considered · C.10 Duplicate Identification Details

The pipeline balances segmentation quality against computational cost, selecting between Nupunkt and SaT by language while evaluating adaptation and batching tradeoffs. It chooses TextTiling for customizable, computationally viable chunking and uses banded simhashes with exhaustive candidate comparison for duplicate identification.

  • C.8 Segmentation Details: SaT generally segments better than Nupunkt, but requires vastly more compute; Nupunkt is therefore selected for languages meeting a 0.6 F1 cutoff.The cutoff was based on F1 scores for identifying SaT sentence-start indices, with English near the threshold.
  • C.8 Segmentation Details: Per-book adaptation makes Nupunkt agree slightly more with SaT, with the models agreeing approximately 93% of the time overall.Manual checking suggests the remaining 7% difference reflects more accurate segmentation, although adaptation may be omitted when speed is prioritized.
  • C.8 Segmentation Details: Base-language Nupunkt runs about twice as fast as SaT, while omitting per-book adaptation reduces estimated full-collection segmentation time from approximately 2 weeks to 3 days on a DGX Spark.Up to 8 simultaneous Nupunkt inference processes were observed before memory bandwidth or RAM limitations; SaT alone was estimated at about 1 month on one NVIDIA A100.
  • C.8 Segmentation Details: Alternative segmenters offered distinct tradeoffs: mwtokenizer was faster but inconsistent cross-lingually, sentencex was accurate but too slow, and pySBD was over 10× slower than Nupunkt.The default English NLTK segmenter was less reliable than Nupunkt, while language-specific NLTK libraries were not exhaustively tested.
  • C.8 Segmentation Details: SaT books were not batched because individual long books can saturate a GPU, although small tests found groups of 200 improved throughput between 15 and 25 percent.SaT internally splits long inputs into overlapping 512-token windows and batches those tokens.
  • C.9 Alternate Chunking Algorithms Considered: GPU-based chunking methods did not fit the available compute constraints, and thresholding sentence-end probabilities failed experimentally as a paragraph-chunking strategy.TextTiling and C99 were computationally viable and customizable with modern multilingual sentence embeddings.
  • C.9 Alternate Chunking Algorithms Considered: TextTiling was chosen over C99 because it produced better duplicate detection later in the pipeline, despite both algorithms being implemented in the repository.Both methods were described as performant and customizable when combined with modern multilingual sentence embeddings.
  • C.10 Duplicate Identification Details: Duplicate candidates are found by splitting each 128-bit simhash into 6 bands and exact-matching bands, while treating pairs within Hamming distance 5 as duplicates.This guarantees every qualifying pair shares at least one band; exhaustive comparison uses two xors, two popcounts, and one addition over 64-bit limbs.

C.11 Bits-Per-Byte Details

The BPB computation uses Qwen3-0.6B with standard Transformers and PyTorch calls, while evaluating proxy features for perplexity prediction across model sizes. Results showed limited general prediction after outlier removal, and BPB values remained unresolved for paragraphs outside the supported length range.

  • Implementation: BPB computation wraps Qwen/Qwen3-0.6B-Base tokenizer and model calls in Transformers and PyTorch, using right padding during batching.Qwen 3 models require right rather than left padding.
  • Determining the Language Model: 3067 random IB-HL pages supported Ridge regression experiments predicting Qwen 3 perplexity across models from 0.6B to 14B parameters.The models used an 80-20 train-test split.
  • Determining the Language Model: Rare bigram ratios and all KenLM models predicted extremely high perplexity, but removing the 1% outliers left almost no correlation between actual and estimated perplexity.Endmatter classification performed poorly because its bimodal behavior weakly tracked only extreme perplexity.
  • Determining the Language Model: N-gram model predictions correlated strongly with one another but not with actual perplexity, while the 0.6B model strongly predicted the 14B model.This behavior was unexpected because similar parameter counts were expected to correlate more strongly.
  • Error Rates in Computing BPB: 4 too-short and 12,391 too-long paragraphs retained unrevisited BPB values because long-paragraph computation could require chunked or fused cross-entropy or multiple GPUs.The alternative implementations were identified after the full pipeline had run.

D Examples with Known Limitations

The pipeline has known unintended side effects despite conservative cleaning, especially for mathematical text and bibliographic citations. These limitations can alter meaning, remove references, and obscure provenance.

  • Known limitations: The pipeline’s conservative cleaning still produces unintended side effects in known cases.The paper presents two examples of such effects.
  • Dehyphenation and Mathematics: Technical mathematics can be meaningfully altered because dehyphenation treats mathematical notation like prose.Minus signs and fraction bars may be interpreted as end-of-line hyphens and removed.
  • Dehyphenation and Mathematics: These mathematical cases are rare among billions of correct merges but may disproportionately affect technical and mathematical volumes.The example changes three hyphenation patterns, each affecting the source OCR’s meaning.
  • Removing Citation Footers: 137 citations to Kriegs-Archiv in source OCR decrease to 64 in output when repeated citation footers are mistaken for running footers.Header/footer removal targets repetitive lines near page tops or bottoms, which can include abbreviated citations.
  • Removing Citation Footers: Marginalia are grouped with main text, placing bibliographic references inside paragraphs and preventing accurate provenance tracking.Restoring these paratextual elements is identified as a future-work goal.

E Using the Parser Library

This section presents a lightweight, pure-Python, dependency-free parser library for filtering and iterating through the annotated dataset. Its BookDataset interface accepts diverse dictionary iterables and supports filtering at book, paragraph, and section levels.

  • Library interface: The parser library uses Python’s standard-library html.parser.HTMLParser to process the final annotated dataset.It is implemented as a lightweight, pure-Python, dependency-free library and tool.
  • Library interface: BookDataset accepts any iterable of Python dictionaries whose schema matches the dataset fields.Inputs may include manually curated dictionaries, partial pipeline outputs, or the streamed HuggingFace dataset.
  • Filtering: Book-level filtering supports ISO 639-3 language selection and minimum or maximum token-count ranges.The language filter accepts one language string or a list, while token-count bounds accept integer arguments.
  • Filtering: Paragraph-level filtering supports language, duplicate exclusion, and floating-point BPB minimum or maximum thresholds.Section filtering follows the same semantics and applies each filter to paragraphs within the section.
Loading 2608.19026v1…