Source-linked AI summary
Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics
Luca Foppiano
TL;DR
PDF corpora report token totals but publish rates per document without decomposing those totals. This paper compares both units and measures truncation recovery with two extractors, finding that 23.06% of documents held 63.08% of the text.
Problem
PDF corpora report token totals but lack token-weighted rates and decompositions of corpus text by length or quality.
Method
The paper compares document- and token-weighted corpus statistics, reconstructs truncated files, and extracts intact and truncated versions with two independent engines.
Results
23.06% of documents held 63.08% of the corpus’s text, while PyMuPDF and PDFium recovered 11.4% and 1.4% of truncated tokens.
Takeaways & Limitations
Corpus statistics should be reported in both documents and tokens, and size-based filters should be evaluated by the text they select.
Takeaways & Limitations
Recovery estimates cover only 67% of truncated documents successfully processed by both extractors, are not based on a random sample, and have no confidence intervals.
Abstract
from arXiv · showhide
PDF corpora advertise their size in tokens but compute every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) per document, and none decomposes its token total. The two units diverge sharply. On CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M web PDFs, 32.6B tokens), 3.02% of text-bearing documents hold half the tokens (Gini 0.807); documents over 50 pages are 5.00% of the corpus but 53.53% of its text. The PDFs produced by a TeX{} toolchain are 1.66% of documents and 4.05% of the text. The clearest casualty is Common Crawl's truncation cap: it affected 23.06% of documents and 63.08% of the text. Reconstructing the truncated files and extracting both versions, two widely used libraries recover 11.4% and 1.4% of that text; between 72% and 97% of affected documents yield nothing; roughly 55--62% of the corpus's text is lost. Under the 5 MiB cap adopted in March 2025, 30.19% of tokens would still be truncated, and recovery on those documents rises only from 3.3% to 13.2%. We recommend that corpus statistics be reported in both units: documents and tokens.
1 Introduction
PDF corpora headline token totals but report rates per document without decomposing text by length or quality. This unit mismatch matters because web-PDF text is highly concentrated, making document-weighted statistics misleading.
- Motivation and contributions: Every headline rate is reported per document, although corpora such as FinePDFs route 368.8M of 1.29B files to OCR.The cited corpus also reports re-fetch recovery and other outcomes in document units.
- Motivation and contributions: ∼5.3k character median and ∼68k 95th percentile in FinePDFs indicate more extreme PDF length skew than HTML, yet its rates remain document-based.FinePDFs contrasts these values with ∼11–13k characters elsewhere.
- Motivation and contributions: Gini 0.807 shows that text mass in web PDFs is far more skewed than document counts suggest.The paper presents this as its concentration result.
- Motivation and contributions: 23.06% of documents but 63.08% of tokens are affected by Common Crawl’s truncation cap, with at most 11.4% recoverable.The paper identifies this as its first measurement of the cap’s text cost.
- Motivation and contributions: 30.19% remains lost at the new 5 MiB limit, underscoring the need to report corpus statistics in both documents and tokens.The proposed analysis directly compares document- and token-weighted headline statistics.
2 Related Work
Prior PDF-corpus work reports totals and processing rates primarily in documents, often without token counts or coverage measures. Truncation is recognized, but its cost in text remains unquantified despite PDFs being disproportionately affected.
- Corpus statistics: FinePDFs reports OCR-routing performance and stage rates in files, while CCpdf reports documents per processing step and language without a corpus token count.FinePDFs reports F1 0.71 on the OCR class.
- Coverage reporting: PDFA selects documents using size and rendering-time thresholds without publishing drop rates, while GovScape and extraction benchmarks omit quantified exclusion or corpus coverage.PDFA discards files over 100 MB or slower than 500 ms to render; GovScape excludes pages lacking embedded text, and extraction benchmarks evaluate only documents that parsed.
- Truncation: 0.8% of successfully fetched records were PDFs but they occupied 11.85% of WARC storage in CC-MAIN-2023-06, illustrating disproportionate PDF truncation exposure.Common Crawl’s 1 MB cap was described as relatively low for PDF files, prompting CCpdf to re-download truncated documents.
- Truncation: 6.8% of PDFs remained truncated after the cap rose from 1 MiB to 5 MiB, versus 0.14% of all types and 0.04% of HTML.PDF truncation remained roughly 49× the general content rate, while the text cost of truncation was not reported.
3 Data and Method
The study deduplicates Common Crawl PDF metadata into 7,932,654 unique documents and measures text mass with published Apache Tika token counts. It also uses provenance fields and refetched originals to identify truncation and recover each document’s true size.
- Corpus: 7,932,654 unique documents and 32,570,135,761 tokens remain after deduplicating 8.3M URL records by post-SHA-256 file identity.The corpus contains every PDF found in CC-MAIN-2021-31, including payloads that Common Crawl truncated and that were refetched whole from origin.
- Corpus: 53.53% of text belongs to documents over 50 pages, although they comprise 5.00% of the corpus when counted purely by page count.The page-count row excludes 0.49% of documents with unparseable page metadata, and orientation takes precedence in the mutually exclusive categories.
- The unit: tika_eval_num_tokens supplies the text mass as a whitespace/ICU token count from Apache Tika 2.8.0 with Tesseract disabled.The analysis claims proportions rather than dependence on an LLM tokenizer; the field is right-censored at 10,000,000, reached by only two documents.
- Truncation: ∼1.049 MB is the observed WARC record length for truncated records, whose provenance fields identify truncation and whose refetched originals provide true document sizes.cc_truncated='length' coincides exactly with fetched_status='REFETCHED_SUCCESS'; the pairing includes fetched_length and the WARC byte range per file.
4 Results
The corpus’s text is highly concentrated in a small minority of documents, so document-count statistics misrepresent the text distribution. Common Crawl truncation disproportionately affects that text, and recovery depends sharply on the extractor and cap.
- Text concentration: 53.53% of the text comes from documents over 50 pages, although they comprise 5.00% of the corpus; 3.02% of documents hold half the tokens.Documents with three pages or fewer are 46.2% of documents but contribute only 6.46% of the text.
- Producer mix: 4.05% of tokens come from TeX-toolchain PDFs, despite those PDFs comprising 1.66% of documents.PowerPoint shows the reverse pattern: 0.82% of tokens versus 2.24% of documents.
- Truncation exposure: 63.08% of the corpus’s text was in the 23.06% of documents truncated at Common Crawl’s 1 MiB cap.At 5 MiB, exposure falls to 30.19% of tokens and 5.94% of documents.
- Extractor-dependent recovery: 11.4% of tokens are recovered by PyMuPDF versus 1.4% by PDFium on identical truncated inputs.The engines recover 20.7% versus 1.9% of pages, while agreeing within 1.7% on intact versions, making extractor choice decisive.
- Failure modes: 72.4% of truncated files yield no text under PyMuPDF, while PDFium fails to open 96.7% and has a 96.8% zero-yield rate.PyMuPDF often opens damaged files by rebuilding cross-reference tables, but content streams may lie beyond the truncation cut.
- Higher-cap recovery: 13.2% of exposed tokens are recovered at 5 MiB versus 3.3% at 1 MiB with PyMuPDF; PDFium rises from 0.1% to 2.4%.Despite the larger prefix, 77.6% of documents still yield no text under PyMuPDF and 97.0% under PDFium.
5 Conclusion
The corpus appears fundamentally different when measured in documents versus text, with category shares diverging by up to 11× and Common Crawl’s cap disproportionately removing text. Corpus statistics should therefore report both units, and size-based filters should be evaluated in text because that is where the corpus’s size lies.
- Conclusion: 11× is the maximum divergence between category shares measured in documents and text.The conclusion characterizes CC-MAIN-2021-31-PDF-UNTRUNCATED as two different corpora under the two counting units.
- Conclusion: 23% of documents but 55–62% of the text was affected by Common Crawl’s cap.The cap therefore imposed a substantially larger burden when measured in text than in documents.
- Conclusion: Every rate in this literature is per-document or unreported, so corpus statistics should be reported in both documents and text.The conclusion specifically names coverage, filter drop, and OCR routing as examples of such rates.
- Conclusion: Size-based filters must be priced in text because size is where the text is.The passage introduces PDFA’s 100 MB and 500 ms cuts as examples of size-based filtering.
Limitations
Corpus-composition and truncation-exposure figures use all 7.9M documents, but recovery results cover only the 1.2M truncated documents successfully processed by both extractors. Technical failures concentrated in ten PyMuPDF shards, making the recovery subset non-random and leaving confidence intervals unreported.
- Coverage of reported figures: 7.9M documents underlie every corpus-composition and truncation-exposure figure, whereas recovery figures cover 1.2M truncated documents successfully processed by both extractors.The 1.2M comprise 67% of the 1.83M Common Crawl truncated documents.
- Extractor limitations: 10 of 64 PyMuPDF shards produced nothing because of technical extractor errors, including out-of-memory failures and hangs.Because shards partition the corpus by archive index, this loss is arbitrary with respect to document content.
- Inference limitations: About a percentage point separates per-shard recovery rates, but the processed documents are not a random sample and confidence intervals are not reported.The reported recovery subset is therefore constrained by technical shard failures rather than documented sampling design.
Data and Code availability
The study uses data from the Digital Corpora et al. (2021) resource, and its code is publicly available on GitHub.
- Data and Code availability: The data was collected from the resource provided by Digital Corpora et al. (2021), and the code is available at the linked GitHub repository.Repository: https://github.com/lfoppiano/cc-wacky-pdf.
A Recovery by document size
Recovery gains from raising the cap are analyzed by document size, using paired measurements on the same documents. The benefit is concentrated immediately above the cap.
- A Recovery by document size: Table 3 breaks paired recovery measurements down by document size.The breakdown corresponds to the paired recovery measurement of §4.6.
- A Recovery by document size: 316,174 documents are cut at both cap points, and the benefit is concentrated immediately above the cap.The table reports PyMuPDF gains in percentage points.