Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale

• IB-HL contains 242B tokens of digitized Harvard Library materials and emphasizes provenance, collection-level deduplication, OCR analysis, legibility improvements, and rich metadata.The dataset was introduced in 2025 as Institutional Books: Harvard Library (IB-HL)1.Source: §1 · p. 3
• Standard internet-oriented pipelines can remove multilingual content, fragment long documents through naive deduplication, and discard paratext such as titles, footnotes, indices, and publisher notices.Excluding non-English content would discard half of IB-HL and narrow its cultural scope; cited examples include (J. Kaplan et al. 2020; Gunasekar et al. 2023; Muennighoff et al. 2023).Source: §1 · p. 3
• Enriched Text provides a configurable dataset by layering machine-readable annotations for paratext, metadata, detected languages, duplicate paragraphs, and front- and back-matter separation over OCR text.HTML-like attributes can be parsed with standard libraries, allowing users to tailor representations to their needs.Source: §1 · p. 3
• The project targets three foundational construction problems: multilingual OCR cleaning that preserves structure, semantically meaningful segmentation respecting document boundaries, and collection-wide duplicate-segment identification.IB-HL-ET extends this work into modern information-access workflows, training pipelines, and computational scholarship.Source: §1 · pp. 3–4

Contributions

• The project releases IB-HL-ET, a public dataset with 217B o200k_base tokens spanning 7B sentences, 1.39B subtopic paragraphs, 297M subtopic sections, and 983k volumes.The dataset is sourced from Harvard Library’s Institutional Books collection.Source: §2 · p. 4
• The project provides a detailed technical report documenting the processing used to create IB-HL-ET.Source: §2 · p. 4
• The release includes the Python pipeline used to produce the collection and a dependency-free Python library for parsing and adapting it to specific use cases.Source: §2 · p. 4
• The project also releases synthetically generated training data for an endmatter classifier and subclassifier.Source: §2 · p. 4

• IB-HL-ET organizes existing OCR as a text-to-text process, making the collection easier to read, filter, search, and study computationally without re-OCRing volumes.This downstream design can augment computationally expensive prior OCR processes without discarding or recreating their outputs.Source: §3 · p. 4
• Annotations separate linguistic content from paratext and distinguish endmatter from middlematter, allowing users to retain, filter, or remove layers for different research needs.For example, users can obtain long-context English prose by filtering annotations rather than altering the underlying text.Source: §3 · p. 4
• The pipeline favors marking and annotating over rewriting or deletion, but removes page numbers and running headers and footers because they become recurring noise after page boundaries are removed.These marginalia retain navigational and bibliographic value, but broader treatment of marginalia is left for future work; unmodified OCR remains available in IB-HL.Source: §3 · p. 5
• The techniques target broadly reproducible, frugal computation, with most processing using CPUs or small single-GPU models and an approximate one-month target on one NVIDIA DGX Spark.Computing bits-per-byte for all paragraphs is the major exception, requiring months of dedicated GPU time even with a 0.6B model.Source: §3 · p. 5

4.1.1 Methodology

• The pipeline shards volumes by primary language category, enabling most steps to run independently and allowing Nupunkt-compatible shards to process without GPUs while other shards require GPUs.It groups up to 200 volumes per shard and uses a distilled static embedding model so embeddings can be computed on CPUs, trading some precision and accuracy for efficiency.Source: §4.1.1 · pp. 6–7, §4 · p. 6

4.1.2 Results

• The pipeline trains 388 base models and two endmatter classification models, including n-gram models for 250 languages and Nupunkt models for 138 languages.Models are trained from up to 30 books per language when available; only 82 languages had the full 30 books, while 168 used every available book in IB-HL.Source: §4.1.2 · p. 7
• The endmatter classifier achieved 0.97 accuracy and F1 0.97, while the endmatter subclassifier achieved 0.90 overall accuracy.The classifier separates frontmatter, middlematter, and backmatter; the subclassifier identifies TOC_INDEX, BIBLIO, and OTHERENDMATTER, using a Model2Vec-distilled multilingual embedding model and synthetic training data.Source: §4.1.2 · pp. 7–8, §4.4 · p. 9, §4.4.1 · p. 10

4.2.1 Methodology

• Soft normalization preserves visible text while removing encoding inconsistencies, whereas hard normalization is transient, lossy, and used only when typographic robustness matters for downstream comparisons.All IB-HL-ET output is soft-normalized; hard normalization supports dehyphenation, exact duplicate-page removal, and duplicate-passage identification but is never presented to users.Source: §4.2.1 · p. 8, §4.2 · p. 8

4.2.2 Results

• Soft normalization of all text required 46h across 1-core jobs, averaging approximately 0.17 seconds per volume.The approach prioritizes accountability and provenance by conservatively normalizing text rather than correcting individual OCR mistakes.Source: §4.2.2 · p. 8, §4.2 · p. 8

4.3.2 Results

• Page-level deduplication found that approximately 81.6% of books had no duplicate pages, while the most extreme case retained 132 pages from 692 scans.The method uses 128-bit simhashes over qualifying 9-grams, a 6-bit near-duplicate threshold, and union-find grouping within each volume.Source: §4.3.2 · p. 9, §4.3.1 · p. 9

4.4.1 Methodology

• Endmatter separation is conservative and non-destructive: detected frontmatter and backmatter are output largely unprocessed, while the remaining pipeline primarily processes middlematter.On average, each book has approximately 14 endmatter pages, and endmatter classification required 211h; errors more often move endmatter into middlematter than the reverse.Source: §4.4 · p. 9, §4.4.1 · p. 11, §4.4.2 · p. 11

4.5.2 Results

• 2,322,246,287 hyphens were removed, with approximately 89% of estimated end-of-line hyphens removed, 7% retained, and 4% replaced by a space while retaining the hyphen.Dehyphenation ranks merging, retaining, or spacing the hyphen using base and book-specific character n-gram models, whose probabilities are weighted equally.Source: §4.5.1 · p. 12, §4.5.2 · p. 13
• End-of-line hyphen removal is most frequent in Cyrillic, Latin, and European languages: English averages 4.9 merged hyphens per page, while Chinese, Japanese, and Arabic average below 0.1.Latin has the highest per-page rate, consistent with dense scholarly printing of classical text.Source: §4.5.2 · p. 13

4.6.1 Methodology

• Running headers and footers are detected as locally concentrated near-duplicate lines among the top or bottom five page lines and removed only after at least three matches within five pages.MinHash signatures over character 5-grams use a Jaccard threshold of 0.85; candidates shorter than six characters are discarded, and conservative thresholds limit excess removal.Source: §4.6.1 · p. 14

4.6.2 Results

• 272,583,638 header and footer lines were removed from 760,238 volumes (77.3%), averaging 277 removals per volume; detection was less accurate for vertically oriented languages.The process required 927h, averaging 3.4s per volume, and language-specific OCR placement errors appear to explain the lower accuracy for vertical scripts.Source: §4.6.2 · p. 14, Table 5 · p. 14

4.7 Page Number Removal

• Page-number removal used OCR lines in the top or bottom five positions, at most eight characters, containing a numeric character and no more than one nonnumeric character, averaging under 0.01 s/volume and requiring 2.5h total.The median removal was close to one number per page, while 13k books had at least three removals per page and the maximum was 8.7 removed “page numbers” per page.Source: §4.7.1 · p. 15, §4.7.2 · p. 15

4.8 Sentence Segmentation

• Sentence segmentation routes 97.9% of books through the Nupunkt engine and the remaining 2.1% through the Sa T engine; Nupunkt is used for languages with English-like punctuation.Per-book Nupunkt adaptation improves performance but requires approximately 16× more compute than using only the base model; Sa T requires a GPU.Source: §4.8 · p. 16, §4.8.1 · p. 17

4.9.1 Methodology

• 1,568h of topical chunking produced subtopic paragraphs using sentence-embedding similarity valleys, with paragraph boundaries selected greedily above a per-book threshold while enforcing a 3-sentence minimum.The implementation averages embeddings over neighboring five-sentence windows, smooths gap scores, and applies a TextTiling-style valley-depth procedure; it runs in O(T+Nlog N+B^2).Source: §4.9.1 · pp. 18–19, §4.9.2 · p. 19

4.9.2 Results

• Subtopic paragraphs average about 4.8 sentences in Japanese and 5.1 in Latin, while 950 paragraphs exceed 100k characters, mostly because of poor OCR on tables.Paragraph lengths are roughly log-normal and unimodal, with the lower bound reflecting the 3-sentence constraint; Nupunkt- and Sa T-segmented volumes have no significant difference in sentence counts.Source: §4.9.2 · p. 19, Table 10 · p. 19

4.10 Duplicate Identification

• Duplicate subtopic paragraphs are identified with 128-bit simhashes over entropy-filtered normalized 9-grams, treating signatures within 5 bits as duplicates and annotating matches instead of deleting them.The method uses six-band indexing to avoid pairwise comparison across approximately 1.4B paragraphs, while library boilerplate is an explicit exception in which two clusters are removed.Source: §4.10 · p. 20, §4.10.1 · pp. 20–21

4.10.2 Results

• The pipeline detected 1,932,557 same-book duplicate clusters, while most duplicate clusters spanned multiple volumes.The average cluster contained ≈2.42 duplicates, but 85 clusters exceeded 1,000 duplicates and were usually missed boilerplate or poorly OCRed tabular filler.Source: §4.10.2 · pp. 22–23
• English had a duplication rate more than 2.5 times that of the next language, while about 43% of books had no duplicate paragraphs and 80% had fewer than 5%.The 24 books with 98–100% duplication were reprints, edition changes, or duplicate books in the collection.Source: §4.10.2 · pp. 22–24

4.11 Bits-Per-Byte

• BPB normalizes language-model negative log-likelihood by UTF-8 bytes, yielding a more tokenizer- and language-independent predictability measure than perplexity.Lower BPB indicates more predictable text, while higher BPB indicates more surprising text; the measure is intended to support paragraph ranking, book-level quality summaries, and downstream filtering.Source: §4.11 · p. 24, §4.11.1 · p. 24

4.11.1 Methodology

• BPB uses Qwen/Qwen3-0.6B-Base with length-grouped batches, but excludes paragraphs of ≤4 characters or over 14,134 tokens and interprets scores relatively within books.Genre, technical material, mathematical notation, and low-resource languages can produce high BPB without indicating low value, so book-level distributions are preferred to corpus-wide cutoffs.Source: §4.11.1 · pp. 24–25

4.11.2 Results

• Using ≈2000 GPU-hours, BPB scores were produced for 1,393,997,373 of 1,394,009,768 released subtopic paragraphs, while 12,395 retained the −1 sentinel.Of the sentinel cases, 12,391 were longer than the 14,134-token limit and 4 were too short.Source: §4.11.2 · pp. 25–26

4.12 Final Annotation and Formatting

• The final output provides HTML-escaped frontmatter_gen, middlematter_gen, and backmatter_gen strings whose tags encode BPB, language, duplicate-cluster, and representative-paragraph metadata.Stripping tags recovers volume text, while attributes support filtering and structured reasoning; the release also includes 21 volume-level metadata descriptors.Source: §4.12 · pp. 26–28

4.12.1 Methodology

• Language detection runs on every subtopic paragraph with polyglot, using neighboring positive detections to infer a matching language for short undetected paragraphs.This paragraph-level approach provides more granular language distributions than chunk-level detection, and the fallback applies below 30 o200k_base tokens when both neighbors agree.Source: §4.12.1 · p. 29
• The pipeline computes token counts and tokenizability with tiktoken using o200k_base, targeting 1.25 tokens per word, and derives linguistic counts with polyglot and its segmentation outputs.Bigrams and trigrams are computed in pure Python over polyglot word lists.Source: §4.12.1 · p. 29

4.12.2 Results

• Metadata computation required approximately 150h of combined wall time on a 16-core MacBook Pro M4 Max.Exact timing was not logged, and the language-count results are reported in Table 17.Source: §4.12.2 · p. 29
• Paragraph-level language distributions closely matched book-level distributions for large languages, while Latin accounted for 3.70% of paragraphs but only 2.20% of books.The higher paragraph share was attributed to Latin quotations or passages in many non-Latin books.Source: §4.12.2 · p. 29
• 98% agreement was found between IB-HL’s primary-language labels and the pipeline’s most common paragraph-level language per volume.Disagreements tended to involve bilingual volumes or language pairs with substantial n-gram overlap.Source: §4.12.2 · pp. 29–30

5 Dataset Analysis

• The three analyses use separate annotation layers to quantify cleaning, measure cross-language duplicate reuse, and relate paragraph BPB to publication metadata.This framework is intended both to validate pipeline components and to facilitate research on the IB-HL-ET collection.Source: §5 · p. 30

5.1 Tokenizability in IB-HL and IB-HL-ET

• Tokenizability increased for every language, with the largest gains in Occitan (+9.5), Persian (+5.8), and Arabic (+5.6).IB-HL-ET also has approximately 26B fewer characters, at least 5-percentage-point lower unique bigram and trigram ratios, and approximately 6-point higher tokenizability than IB-HL.Source: §5.1 · pp. 30–31

5.2 Multilingual Reuse of Duplicated Text

• Only 5.3% of 124,049,981 duplicated paragraphs have detected languages differing from their host volumes, falling to 2.3% among paragraphs at least 200 characters long.English, German, French, and Russian duplication is 95–99% same-language-bound, whereas Latin and Greek frequently occur in texts with different primary languages; short fragments also produce detection bias.Source: §5.2 · pp. 31–32

5.3 Bits-Per-Byte Over Time

• Median per-volume BPB declines from approximately 2.5 in the 17th century to approximately 1.4 in the mid-19th century and approximately 1.0 by the mid-20th century.The English subset follows approximately the same trajectory with a slightly lower median, but BPB also reflects OCR noise and post-1930 public-domain sampling bias.Source: §5.3 · pp. 32–33

Discussion and Further Directions

• Annotation over removal makes IB-HL-ET adaptable for new use cases and supports more robust, computationally addressable, cross-compatible text corpora.The authors propose standardized enriched-text formats as a way to improve AI training data practices and enable future annotations or analyses.Source: §6 · p. 33

Analysis of the Enriched Text

• Marked duplicate paragraphs and per-paragraph BPB enable research on cross-lingual textual reuse, historical language change, OCR degradation, and training-data availability.These analyses expose research questions that a flattened token stream would conceal, including language-specific writing trends and era-specific OCR effects.Source: §6 · p. 33

Pipeline Improvements

• The current dataset remains limited by weaknesses in recognizing tables, marginalia, and technical language, which future OCR advances may improve.The pipeline was built for Harvard’s OCR, and other collections or OCR systems may produce different failure modes and engineering challenges.Source: §6 · pp. 33–34
• The unusually low BPB in the 1660s reflects 58 high-quality, formulaic legislative volumes, while the other 21 English volumes average approximately 1.64 BPB.OCR confidence suggests the broader English curve is relatively consistent rather than merely OCR noise, although disentangling the variations requires further research. [22] [23]Source: §6 · p. 33

Towards a Virtuous Cycle

• Collaborative extensions to the pipeline and annotations aim to make more books computationally accessible and improve low-resource-language models while preserving provenance, reproducibility, and auditability.The proposed cycle incorporates new cleaning methods, corrections, annotations, and perspectives into source material for further collections.Source: §6 · p. 34

• The collection deliberately includes volumes identified by HathiTrust as “public domain,” “public domain in the United States,” or “CC-Zero.”These criteria are intended to reduce risk but do not eliminate uncertainty in copyright determinations.Source: §8 · p. 35
• Some volumes may be public domain in the United States but protected elsewhere, and missing rights information does not guarantee public-domain status.Rights information reflects available data at determination time and is provided in good faith without guarantees of completeness or accuracy.Source: §8 · p. 35
• Users are solely responsible for independent legal assessments, permissions, and potential trademark, privacy, publicity, or other rights restrictions.Rights holders may contact the project about misidentified or improperly included works for review.Source: §8 · p. 35

9.1 Harmful Language and Content in this Dataset

• Historical works may contain outdated, offensive, harmful, or inaccurate language and perspectives, so they should be approached with critical awareness and cultural sensitivity.The materials are retained in original form for historical context and research, but they are not a stand-alone collection designed to provide balanced perspectives.Source: §9.1 · p. 35

9.2 Harmful Language in Bibliographic Description

• Bibliographic metadata may reproduce harmful, outdated, or biased terminology because descriptions, standardized terms, and transcribed book language can reflect historical and institutional biases.Metadata may also omit important perspectives or use language that differs from terms currently preferred by depicted communities.Source: §9.2 · pp. 35–36

9.3 Generated and Experimental Content

• Generated and experimental content is provided as is, may contain errors or inaccuracies, and should be independently verified using users’ own judgment.The statement notes that reasonable care was taken to assess quality but does not provide warranties.Source: §9.3 · p. 36

A Dataset Fields

• Dataset fields distinguish source-derived (_src) from pipeline-generated (_gen) values, with text statistics computed over middlematter.These conventions define the dataset schema and the scope of its text statistics.Source: Table 23 · p. 44

B Additional related work

• The work situates its design within research on dataset cleaning, OCR analysis, and neural OCR correction, including ICDAR competitions where neural and BERT-based models outperformed most alternatives.The cited literature spans inexpensive boilerplate detection, statistical OCR methods, surveys, and modern transformer-based techniques (Weninger, W. H. Hsu, and Han 2010; J. Wang and J. Wang 2015; Kohlschütter, Fankhauser, and Nejdl 2010; Kukich 1992; Tong and Evans 1996; Jatowt et al. 2019; T. T. H. Nguyen et al. 2021; Chiron et al. 2017; Rigaud et al. 2019; Devlin et al. 2019).Source: B Additional related work · p. 45
• The appendix contrasts this approach with multistage curation pipelines such as C4, which use heuristic cleaning, English restriction, and chunk-based deduplication on web-scraped data.Data curation can include sourcing, deduplication, cleaning, and filtering.Source: B Additional related work · p. 45
• The related work identifies opacity in language-model training data and notes DCLM’s benchmark framework for improving transparency and comparison (J. Li et al. 2024).The discussion names LLaMA, Mistral, Gemma, GLM, and Kimi as models revealing little about underlying training data (Touvron et al. 2023; Jiang et al. 2023; Gemma Team et al. 2025; GLM-5-Team et al. 2026; Kimi Team et al. 2025).Source: B Additional related work · p. 45
• The approach was shaped by the progression from large-scale data curation through filtering and extraction to post-creation analysis, alongside Datacomp-LM’s literature survey (Gao et al. 2020; Raffel et al. 2020; Wenzek et al. 2020; Dodge et al. 2021; J. Li et al. 2024).The appendix presents this material as additional related work informing the design.Source: B Additional related work · pp. 44–45

C Details on Processing

• The processing appendix supplies additional technical details and design rationale for the pipeline.It frames the following implementation choices as part of the processing methodology.Source: C Details on Processing · p. 45

C.1 Preprocessing Details

• Language-stratified base models are updated for each book, and small model or pre-cleaning noise generally has limited impact because most text is sufficiently clean.The setup relies on IB-HL language reporting for stratification; this behavior holds for most languages and volumes, though noise remains a source of variation.Source: §C.1 · p. 45
• Shards of 200 books are split into Nupunkt-compatible and incompatible languages so CPU-sufficient sentence segmentation is separated from GPU-dependent processing.After this partitioning, all further processing except deduplication is independent across the resulting shards.Source: §C.1 · p. 45

C.2 Unicode Normalization

• Soft normalization applies NFC while preserving line breaks, curly quotes, hyphens, dashes, ligatures, and accents; hard normalization applies NFKC and produces a single line with ASCII punctuation.Both modes remove zero-width characters U+200B and U+FEFF, normalize Unicode spaces, collapse consecutive spaces, and strip per-line boundary spaces; hard normalization additionally standardizes line breaks, dashes, quotes, soft hyphens, tabs, and newlines.Source: §C.2 · p. 46

C.3 Duplicate Page Removal

• Duplicate-page detection hard-normalizes pages, considers pages with at least 50 non-whitespace Unicode code points, and compares character 9-gram simhashes using a Hamming-distance threshold of 6 bits.The 50-character cutoff prevents excessive removal of short content but mildly biases detection against languages with more semantic content per character; nine-character n-grams balance exact-content sensitivity against random collisions.Source: §C.3 · p. 47
• Duplicate-page simhashing uses a compiled C++ MurmurHash3 implementation with optimized popcount comparisons, achieving an approximately 100-fold speedup over direct Python simhash while making accidental collisions negligible.The implementation is designed for little-endian 64-bit platforms and is verified against mmh3; the six-bit threshold is selected to make random near-collisions rare.Source: §C.3 · p. 47

C.4 Endmatter Separation

• Endmatter processing batches pages from one book at a time, with approximately 1024 pages optimal at 7200 pages/s per CPU core and typical batches near 350 pages at 6800 pages/s per core.Memory bandwidth limits scaling, while the 512MB Model2Vec embedding table is the main bottleneck because it does not fit most CPU caches; the same models are applied across all languages.Source: §C.4 · p. 48

C.5 Dehyphenation

• Dehyphenation combines per-book and base-language statistics because per-book models lack reliable counts for long words appearing only once, while a naive joint estimator with add-k smoothing and fixed Stupid Backoff ranks alternatives effectively.A full Kneser-Ney KenLM model was more precise but added approximately 10s per book; cheaper alternatives nearly always produced the same relative ranking at substantially lower cost.Source: §C.5 · p. 48

C.6 Running Headers and Footers

• Running-header and footer detection uses a Jaccard similarity threshold of 0.85, chosen to allow approximately one typo or one changed page-number digit while remaining conservative in practice.The implementation uses datasketch’s default SHA-1 hashing and locality-sensitive hashing; permutation-and-min operations, rather than hashing itself, dominate timing.Source: §C.6 · p. 50

C.7 Page Number Removal

• Page-number removal identifies numeric symbols by Unicode category, accepting bare, marked, Arabic-Indic, Devanagari, full-width, and Unicode Roman numerals but not dash-flanked numbers.Han ideographic numerals are excluded because treating them as numeric would misclassify legitimate text such as ‘十年’ as page numbers.Source: §C.7 · p. 50
• Page-number removal remains vulnerable to OCR ambiguity because visually similar symbols can have different meanings across scripts, including digits, letters, and punctuation-like circles.The method’s Unicode-based rules therefore inherit OCR-engine decisions and known script-specific gaps, including ASCII-letter renderings of Roman numerals and Han ideographic numerals.Source: §C.7 · p. 50

C.8 Segmentation Details

• A 0.6 F1 cutoff assigns most IB-HL books to CPU-friendly Nupunkt, while SaT generally segments better but requires vastly more compute.The cutoff was chosen from multilingual benchmarks, with English near the threshold; Nupunkt models run about twice as fast as SaT.Source: §C.8 · pp. 50–52
• Per-book Nupunkt adaptation differs from base models on 7% of boundaries and manual checks indicate those changes generally improve segmentation.Base and adapted models agree approximately 93% of the time, while adapted models agree slightly more with SaT.Source: §C.8 · p. 52
• The chosen segmentation split is estimated to process IB-HL in approximately 2 weeks on one DGX Spark, or 3 days without per-book adaptation.A single NVIDIA A100 is estimated to require about 1 month for all-SaT segmentation.Source: §C.8 · p. 52
• Alternative segmenters exposed speed, language, or scalability tradeoffs: sentencex was accurate but slow, pySBD was over 10× slower than Nupunkt, and blingfire was comparable.mwtokenizer25 was faster but inconsistent outside Western European languages; NLTK’s default English segmenter was less reliable than Nupunkt.Source: §C.8 · pp. 52–53
• The pipeline omits SaT batching because long books can saturate a GPU, although batching groups of 200 improved throughput by 15–25% in small tests.The added complexity was not justified for only 20,629 books, so the implementation prioritizes simple error tracking and fault recovery.Source: §C.8 · p. 53

C.9 Alternate Chunking Algorithms Considered

• TextTiling and C99 were computationally viable with multilingual sentence embeddings, but TextTiling was selected because it produced better downstream duplicate detection.A probability-threshold variant of sentence segmentation did not work well, and available GPU-based chunkers exceeded compute constraints.Source: §C.9 · p. 53

C.10 Duplicate Identification Details

• Duplicate candidates are found by splitting each 128-bit simhash into six 21–22-bit bands and comparing pairs sharing a band when their Hamming distance is at most 5.Exact band matches restrict exhaustive comparisons to small buckets; a 30k-item safety limit was never reached for this collection.Source: §C.10 · pp. 53–54
• The duplicate-identification pipeline memory-maps a ≈23GB simhash array, merges confirmed pairs with union-find, and fits within 48GB of RAM.The union-find structure uses roughly 13GB for 1.4B paragraph entries, while workers stream confirmed pairs to a parent process.Source: §C.10 · p. 54

C.11 Bits-Per-Byte Details

• Qwen3-0.6B provides the BPB computation through standard Transformers and PyTorch calls, with right padding required for batched inputs.The implementation uses per-byte normalization rather than per-token normalization.Source: §C.11 · p. 54
• Rare-bigram ratios and KenLM n-gram features predicted extremely high perplexity, but removing the top and bottom 1% outliers left almost no correlation with actual perplexity.The examined predictors also included Qwen3 tokenizer n-grams, Endmatter-classifier outputs, embeddings, and basic linguistic descriptors.Source: §C.11 · pp. 54–56, Figure 29 · p. 55
• Endmatter classification predicted perplexity poorly because its bimodal behavior obscured continuous variation, although endmatter weakly tracked extreme perplexity.N-gram model predictions correlated strongly with one another despite using different text units, but not with actual perplexity.Source: §C.11 · p. 56
• Kendall τ=0.782 shows that perplexity rankings were largely preserved, motivating use of the smallest, most computationally efficient model.The study compared Qwen3 models from 0.6B to 14B parameters using Ridge regression on an 80-20 split of 3,067 IB-HL pages.Source: §C.11 · pp. 54, 56
• The pipeline did not recompute BPB for 4 too-short or 12,391 too-long paragraphs, although chunked or fused cross-entropy or multi-GPU computation could support long paragraphs.This limitation was identified after the full pipeline had run.Source: §C.11 · p. 56

Overview

• The authors identify unintended side effects despite conservative cleaning, illustrating limitations that warrant user awareness and future improvement.They present two examples of such effects.Source: D Examples with Known Limitations · p. 56

Dehyphenation and Mathematics

• Dehyphenation can alter mathematical meaning by removing or relocating hyphens, including changing x - 1 - 3 to x - 1 3 and producing -4x-7.All three illustrated changes affect the source OCR’s meaning, although such cases are rare relative to billions of correct merges and may disproportionately affect technical volumes.Source: D Examples with Known Limitations · pp. 56–57

Removing Citation Footers

• Header/footer removal can delete repeated abbreviated citations mistaken for running footers, reducing references to Kriegs-Archiv from 137 source occurrences to 64 output occurrences.The mechanism targets repetitive lines near page tops or bottoms, including citations in footers or marginalia.Source: D Examples with Known Limitations · p. 57
• Marginalia are grouped with main text, placing surviving bibliographic references inside paragraphs and preventing accurate provenance tracking.Restoring these paratextual elements is identified as a future-work goal.Source: D Examples with Known Limitations · p. 57

• The parser library is lightweight, pure Python, dependency-free, and built on the standard library’s html.parser.HTMLParser.It provides a tool for filtering and iterating through the final annotated dataset.Source: E Using the Parser Library · p. 57
• BookDataset accepts any iterable of dictionaries matching the dataset schema, including curated records, partial pipeline outputs, and streamed HuggingFace datasets.This interface exposes the dataset through book, section, and paragraph objects.Source: E Using the Parser Library · pp. 57–58
• Book-level filtering supports primary language and inclusive token-count ranges through optional language, token_count_min, and token_count_max arguments.Language filters accept one ISO 639-3 string or a list of strings, while token-count filters accept integers.Source: E Using the Parser Library · p. 59
• Paragraph filtering supports language, deduplication, and BPB ranges, while section filtering applies the same semantics and omits sections whose paragraphs are all excluded.Paragraph filters accept bpb_min and bpb_max floats, and per-volume BPB values remain available as attributes.Source: E Using the Parser Library · p. 59