CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data

1 Introduction

• Document-understanding models use multimodal document representations, including token positions and page images, and require large pretraining datasets.Document understanding is defined as converting a document into meaningful information.Source: §1 · p. 2
• Existing document-dataset efforts have focused on single domains, while web-wide corpora have focused on text rather than multimodal documents such as PDFs.The paper identifies an all-over-the-Internet approach to multimodal documents as missing.Source: §1 · pp. 2–3
• The paper describes a PDF-corpus pipeline and shares an index and download script for a 14.5-million-page collection intended for 2D language-model pretraining.The corpus may also serve as a source for derived datasets.Source: §1 · p. 3

2 Related works

• Existing document collections are generally single-domain or single-language; the paper aims instead for a diverse, multilingual PDF dataset.A previously presented dataset was not shared, and its paper described processing decisions without analyzing them.Source: §2 · pp. 3–4
• Prior PDF and document collections include domain-specific datasets such as IIT-CDIP, DoRe, Docbank, and Publaynet.The collections described include tobacco-library reports, company annual reports, and scientific articles.Source: §2 · p. 3
• Common Crawl has supported text-focused corpora for multilingual, parallel-text, and question-answering uses, distinct from this paper’s focus on PDF documents.Examples include CCNet, CCMatrix, and CCQA.Source: §2 · pp. 3–4

3.1 Common Crawl

• The May 2022 Common Crawl dump contained 3.45 billion web pages and 462 TB of uncompressed content; the authors selected it to focus on recent documents.The paper notes that applying the procedure to earlier crawls could yield more PDFs and enable diachronic analysis.Source: §3.1 · p. 5
• Common Crawl dumps include PDFs because the crawler treats them as text files, but not image, Excel, or DOCX files.The dumps consist mostly of HTML pages and include an index of crawled pages.Source: §3.1 · p. 5

3.2 PDF links extraction

• The selected CDX method filtered captures by the application/pdf MIME type, while WARC-linked files were less likely to be available in the dump or at their original address.The authors note that processing even a small share of WARC files could still add a significant number of PDF links.Source: §3.2 · p. 5
• The authors compared extracting PDF links from CDX index files with searching raw WARC crawl data, then chose CDX files to speed up the pipeline.CDX files were simpler to download; WARC processing could yield more links but only a limited number of WARC files could be processed.Source: §3.2 · p. 5

3.3 URL-based language detection

• The authors limited the language set to 11 languages and used URL heuristics as a preliminary filter before proper language detection on downloaded contents.For example, PDFs from .pl domains were assumed Polish unless the URL indicated English.Source: §3.3 · p. 6
• Common Crawl’s language metadata was unusable for this task because its detector applies only to plain text or HTML and few PDF links had language metadata.The metadata was added to CDX files in August 2018.Source: §3.3 · p. 6

3.4 Filtering out spam

• The authors characterize PDF files on the Internet as having relatively little spam compared with HTML pages and often containing formal business, legal, or scientific material.Source: §3.4 · p. 6
• A URL-pattern heuristic flagged suspicious PDFs and discarded domains with a large share of such URLs; a manually annotated sample of 1,000 documents contained no spam PDFs.The telltale pattern was a long filename of lowercase letters interspersed with hyphens.Source: §3.4 · p. 6

3.5 PDF data download methods

• At most three PDFs per domain were downloaded for each language, reduced to one for English and two for German to balance diversity and abundance.Source: §3.5 · p. 6
• The provided script can extract individual PDFs directly from a Common Crawl dump without downloading the whole dump.Source: §3.5 · p. 7
• A 1 MB crawler truncation limit means only small PDFs can be extracted from dumps; larger files must be downloaded from their original sites.Source: §3.5 · p. 7
• 94.94% of downloaded documents were successfully processed, according to Table 2’s processing statistics.Source: Table 2 · p. 7

3.6 Born Digital scanner

• Documents with textual content in the PDF structure can avoid OCR: direct extraction supplies tokens and bounding boxes while reducing time and OCR errors.Source: §3.6 · pp. 7–8

3.7 Born Digital detection heuristics

• The detector marks files with visible text length above 100, no hidden text, and no images as not requiring OCR.The statistics were extracted with the authors’ Digital-born PDF Scanner tool.Source: §3.7 · p. 8

3.8 OCR processing

• The DjVu-based tool with a born-digital detector was fastest and produced better output than OCR engines; Tesseract with built-in language detection was slowest and excluded from the final pipeline.The tested alternatives also included Tesseract with URL-based language detection and Azure CV API.Source: §3.8 · p. 9, Table 4 · p. 9

• The final pipeline uses URL-based detection and content-based detection with the langdetect library.Source: §3.9 · p. 10
• 97.3% of documents were correctly assigned a language, although both mechanisms detect only one language and 27 sampled documents contained multiple languages.Source: §3.9 · p. 10
• Content-based detection achieved a 94.21% average F1-score, compared with 90.51% for URL-based detection.Japanese was the exception where content-based detection did not perform better; Arabic and English were the least satisfactory languages for both mechanisms.Source: §3.9 · p. 10
• 98.05% average F1-score was achieved when the proper Tesseract model was used.The paper attributes the increase to language identification operating on the proper alphabet.Source: §3.9 · p. 10

Possible alternatives

• Spacy and langdetect achieved the best similar F1-scores, at 94.33% and 94.21%, respectively; gcld3 was fastest, but langdetect was selected to balance quality and resource use.Source: Possible alternatives · p. 10, Table 6 · p. 11

3.10 Produced index

• The index caps downloads at 200,000 documents per language to balance language representation with dataset size.Statistics for the index are presented in Table 2.Source: §3.10 · p. 11
• The corpus is smaller than prior corpora by total documents and pages, but has the longest average documents, spans 11 languages and many websites, and mostly contains files created after 2010.Source: §3.10 · p. 11