RefinedGAW: A Refined German Academic Web Dataset

• Across HTML and PDF, 88% of German RefinedGAW documents have no near-duplicate candidate match in the three evaluated reference datasets, supporting complementary coverage.Source: §1 · p. 1
• RefinedGAW packages 122 million unique documents totaling 314 billion tokens, derived from the German Academic Web archive.Source: §1 · p. 1
• The German subset contains substantial academic research content, alongside educational, math-heavy, and expert-oriented material, with marketing-heavy content rare.Source: §1 · p. 1
• The queryable metadata index supports fine-grained selection, while retained URLs, crawl occurrences, capture times, and source versions support longitudinal study.Source: §1 · p. 2

2 Dataset Construction

• The archive contains semi-annual snapshots of the public web presence of German academic institutions from 2012 onward, collected from roughly 150 seed domains.The crawls are processed independently, with exact-text deduplication across crawls conducted separately for HTML and PDF.Source: §2 · p. 2

2.1 HTML Text Extraction and Deduplication

• HTML extraction retains texts longer than 300 characters, then global exact-text deduplication reduces 786 million extracted documents to 111 million unique texts.The pipeline retains crawl, capture, URL, and timestamp metadata, as well as links to textually different versions served from the same URL.Source: §2.1 · p. 2

2.2 PDF OCR and Deduplication

• PDF processing first deduplicates byte-identical files, then OCRs pages at 300 dpi and deduplicates again by extracted text, leaving 10.6 million unique texts.The initial stage avoids 69.8 million redundant OCR tasks; the pipeline records page-level outcomes and quality flags.Source: §2.2 · p. 3

2.3 Document-Level Annotation

• GlotLID v3 assigns one language label per retained document, while propella annotates 18 document properties for composition analysis and subset selection.For PDFs, language identification uses the concatenated document text rather than individual page labels.Source: §2.3 · p. 3

3.1 Scale and Format Contributions

• PDFs account for 27% of all tokens but only 9% of documents; 78 billion tokens occur in documents of at least 131,072 tokens.The median PDF text is 1,292 tokens, compared with 337 tokens for HTML documents.Source: §3.1 · p. 3

3.2 Language and Source Composition

• German is the top language label for 45% of documents, while English and German account for 35% and 33% of tokens, respectively.The corpus is not German-only because the GAW has an institutional scope.Source: §3.2 · p. 4
• The ten highest-yield HTML domains contribute 36% of HTML tokens from just 5% of HTML documents, so the corpus is not a uniform sample of German academic institutions.Host and crawl provenance lets users inspect or correct this imbalance when sampling.Source: §3.2 · p. 4

3.3 Annotation-Based Content Profile

• 75% of HTML documents and 91% of PDF texts have good or excellent content quality; PDFs are more often analytical (32% versus 7%) and educational (38% versus 7%), while heavy or pure marketing appears in about 1% of each.Advanced or expert audiences are indicated for 60% of HTML documents and 71% of PDF texts; academic-research or scientific-content labels account for 235 billion tokens, about three quarters of the dataset.Source: §3.3 · p. 4

4.1 Document-Length Comparison

• Beyond roughly 32,000 tokens per document, RefinedGAW-HTML-DE retains more tokens than FineWeb2-DE, including 2.4 billion versus 0.5 billion above 131,072 tokens.FinePDFs-DE retains more than RefinedGAW-PDF-DE at every length threshold; German Commons has the heaviest long-document tail.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 Overlap with Reference Datasets

• 88% of German RefinedGAW documents have no candidate match in the evaluated reference datasets, while matches concentrate within the same source format.Across the complete release, candidate matches occur for 5% of HTML documents and 9% of PDF documents; these are candidate, not verified, matches.Source: §4.2 · p. 5

4.3 Annotation-Based Content Comparison

• Academic research labels cover 60% of RefinedGAW-HTML-DE documents and 66% of RefinedGAW-PDF-DE documents, compared with 3% of FineWeb2-DE and 11% of FinePDFs-DE, respectively.Scientific technical content also has higher prevalence in both RefinedGAW subsets than in their format-matched references.Source: §4.3 · p. 5, Figure 3 · p. 6
• Heavy or pure marketing labels cover 1% and 2% of RefinedGAW-HTML-DE and RefinedGAW-PDF-DE documents, while safety concerns are labeled for 0.1% of each subset.The authors suggest the low safety-concern prevalence most plausibly reflects the institutional crawl scope, since they apply no safety filters.Source: §4.3 · p. 5
• 87.6 billion academic-research tokens in RefinedGAW-DE exceed the 64.9 billion in FineWeb2-DE and FinePDFs-DE combined, despite RefinedGAW-DE having about one fifth as many German-labeled tokens overall.Academic research accounts for 83% of RefinedGAW-HTML-DE tokens and 84% of RefinedGAW-PDF-DE tokens.Source: §4.3 · pp. 5–6
• The profile is not uniformly favorable: RefinedGAW-HTML-DE has lower mostly-or-entirely-content prevalence and reasoning labels than web references, while its PII flag rate is higher.The authors note that content-ratio differences are sensitive to extractor configuration and conclude that RefinedGAW is complementary rather than a higher-quality substitute.Source: §4.3 · pp. 6–7

5.1 Metadata Index and Document Selection

• The release provides document identifiers, URLs, crawl and version provenance, length statistics, propella annotations, and quality scores as a metadata index rather than archived text.The index license covers the index and derived metadata, not the referenced source documents.Source: §5.1 · p. 7
• The accompanying rehydration tool retrieves currently public resources and runs the text-extraction pipeline, while applying current robots.txt rules at retrieval time.The release design reflects uncertainty around retaining and redistributing curated corpora; earlier captures cannot be filtered retroactively for text-and-data-mining reservations.Source: §5.1 · p. 7

5.2 Present-Day Rehydration

• Present-day rehydration downloads 36% of sampled HTML records and 40% of sampled PDF records, while exact reproduction of stored text is rare at 5% and 8%, respectively.For PDFs, 56% of downloaded files are byte-identical to the archived file, but only 36% of those yield the stored text.Source: §5.2 · pp. 7–8, Table 3 · p. 8
• Source availability rises with crawl recency: HTML recovery increases from 19% for February 2013 to 75% for June 2025, and PDF recovery from 18% to 83%.The reported top-quality-group recovery advantage is partly a date-composition effect.Source: §5.2 · p. 8

6.1 Longitudinal Academic Web Archives

• RefinedGAW integrates extracted HTML and PDF text, global exact deduplication, capture provenance, and document annotations into a reusable NLP resource.Earlier GAW work supported longitudinal analysis, but selected-page studies required task-specific access and preprocessing.Source: §6.1 · p. 8

6.2 German and Multilingual Web Corpora

• Unlike German and multilingual corpora built from broad crawls, heterogeneous sources, or translation, RefinedGAW draws on repeated captures from an independently operated academic crawl.Its distinction is access to capture and source history, not another filtering recipe for the same crawl base.Source: §6.2 · p. 8

6.3 Scientific-Document and PDF Corpora

• RefinedGAW extends publication-centered corpora with teaching materials, administrative documents, project pages, staff profiles, and news from institutional websites.FinePDFs provides a large-scale comparison for its PDF component, while RefinedGAW’s focused institutional collection differs from PDFs found in broad web crawls.Source: §6.3 · pp. 8–9

7 Conclusion

• The 26 processed crawls yield 122 million unique annotated HTML and PDF texts totaling 314 billion tokens, with crawl-occurrence and source-version provenance retained.The release provides a metadata index and rehydration software rather than redistributing archived text.Source: §7 · p. 9
• 87.6 billion tokens labeled academic research in RefinedGAW-DE exceed the 64.9 billion tokens across FineWeb2-DE and FinePDFs-DE, whose combined subset is almost five times larger.88% of German RefinedGAW documents returned no near-duplicate candidate match in the three evaluated German reference datasets.Source: §7 · p. 9
• Evaluating selected subsets in language-model training and using retained provenance to study institutional web change remain future work.The paper makes no empirical claim about language-model value or longitudinal findings.Source: §7 · p. 9

Limitations

• The archive omits academic writing outside selected hosts and misses pages within them; snapshot differences may reflect crawl conditions rather than website change.A single crawl covers an estimated half of the crawlable German academic web at the URL level.Source: Limitations · p. 9
• Extraction can omit, reorder, or corrupt text, and the 300-character cutoff removes potentially useful short documents; separate-format exact deduplication leaves near-duplicates and pooled counts may be inflated.The pooled token totals are sums, not a cross-format-deduplicated union, and alternative extraction or deduplication choices were not evaluated.Source: Limitations · p. 9
• Content labels and quality scores lack collection-specific human validation, while differing language-identification methods and cutoffs change the German comparison populations.Token attribution counts a whole document when it has a label, and HPLT 3.0-DE is absent because it was not tokenized.Source: Limitations · p. 9
• The directional overlap analysis covers three selected datasets; its 88% figure means no candidate was returned, not verified absence from all existing corpora.Establishing absence would require a broader, verified comparison.Source: Limitations · p. 9
• The metadata-only release cannot reproduce a fixed full-text snapshot, and rehydration depends on current source availability, robots.txt rules, extractor behavior, and nondeterministic PDF OCR.The paper also leaves language-model training and substantive longitudinal analysis for future work, making no empirical claims about either.Source: Limitations · pp. 9–10

Ethical Considerations

• The index and metadata are released without archived text; rehydration retrieves currently public sources under current robots.txt rules, and users must assess individual sources under applicable terms.The legal basis for public full-text redistribution is unsettled; the metadata index is licensed CC BY 4.0.Source: Ethical Considerations · p. 10
• Propella’s binary PII label can miss personal information and does not anonymize; retrieval should be limited to research needs, with context-specific privacy review and human review where errors could affect people.RefinedGAW is not intended for longitudinal profiling, surveillance, or consequential assessment of people or institutions.Source: Ethical Considerations · p. 10
• Institutional coverage and source-text biases may be reproduced by trained models, while annotations and quality scores are unsuitable for evaluating people, institutions, or documents.Before training, users should review permissions, apply privacy and safety controls, and perform benchmark decontamination; inclusion does not certify training suitability.Source: Ethical Considerations · p. 10

A Token Mass Above Document-Length Thresholds

• HTML and PDF collections retain 185.5 billion tokens (81% of HTML tokens) and 79.9 billion tokens (94% of PDF tokens), respectively; these are extracted-text lengths, not evidence of coherent long-context sequences.Even at 131,072 tokens, they retain 57.7 billion and 20.7 billion tokens, together accounting for 25% of all RefinedGAW tokens.Source: A Token Mass Above Document-Length… · p. 12

B Exact-Text Recurrence

• Across snapshots, 58% of PDF texts versus 32% of HTML texts occur in at least two crawls; within a crawl, repeated captures are more prevalent for HTML than PDF.44,859 HTML texts and 68,001 PDF texts occur in all 26 crawls.Source: B Exact-Text Recurrence · p. 12

C.1 Predicted Language Composition

• German and English together account for 65% of HTML tokens and 79% of PDF tokens under threshold-free top-label assignment; these are model predictions, not human-verified labels.Latin predictions account for 0.3% of HTML documents but 4.3% of HTML tokens, concentrated in a small number of long documents.Source: §C.1 · pp. 12–13

C.2 Sensitivity of the German Subset

• At a 0.4 GlotLID cutoff, HTML retains 51.8 billion tokens and 41.8 billion academic-or-scientific tokens, while PDF retains 36.9 billion and 31.9 billion, respectively.HTML academic-or-scientific share falls from 83.5% under the top-label definition to 65.9% at 0.9; stricter thresholds disproportionately remove token-heavy HTML documents.Source: §C.2 · p. 14, Table 4 · p. 15
• At format-relevant cutoffs, approximately 83% of each RefinedGAW subset has no candidate in its format-matched reference dataset, though the exact overlap rate depends on language selection.HTML candidate rates rise from 11.32% under the top-label definition to 16.53% at 0.8; PDF rises from 15.57% to 17.03% at 0.4.Source: §C.2 · p. 14

C.3 German- and English-Labeled Content

• The English-labeled subset contributes 111.0 billion tokens across both formats, slightly more than the German-labeled subset’s 104.8 billion, but content profiles differ by format and outcome.For HTML, academic-or-scientific content has a higher share and absolute yield in the German subset; for PDF, the English subset has a higher share in every displayed outcome.Source: §C.3 · p. 15, Table 5 · p. 16

D Token Yields by Content Property

• RefinedGAW-HTML-DE has 53.6 billion academic-research tokens versus FineWeb2-DE’s 22.3 billion, while RefinedGAW-PDF-DE has 8.7 billion math-heavy tokens versus FinePDFs-DE’s 5.0 billion.RefinedGAW is smaller than these comparators, and its yields are lower for several other content outcomes; high concentration offsets scale only for selected properties.Source: D Token Yields by Content Property · p. 16, Figure 7 · p. 17

E.1 Source-Host Concentration

• The eight highest-yield HTML hosts contribute 33.4% of tokens but 4.5% of documents; half of HTML tokens come from 37 hosts, versus 162 PDF hosts.The top HTML host alone contributes 9.3% of tokens from 0.09% of documents, so users can cap, reweight, or exclude dominant hosts using provenance.Source: §E.1 · p. 18, Figure 8 · p. 19

E.2 Longitudinal Contribution

• The first-observed share falls from 92.4% of HTML tokens in February 2013 to 40.5% in June 2025, and from 49.4% to 8.3% for PDF.The June 2025 crawl still adds 3.4 billion previously unseen HTML tokens and 1.3 billion PDF tokens; first-observed refers to the archive, not source publication date.Source: §E.2 · p. 19, Figure 9 · p. 20

E.3 PDF Length and Extraction Diagnostics

• Only 3% of PDF documents exceed 100 pages, yet they account for 41% of PDF tokens, making token-weighted extraction diagnostics important.Severe-repetition flags affect 16% of documents but 48% of tokens; flags are non-exclusive pipeline diagnostics, not human-validated error labels, and flagged documents remain included.Source: §E.3 · p. 20, Figure 10 · p. 21

F.1 Candidate Validation and Length Sensitivity

• Only 67% of HTML/FineWeb2-DE candidates and 59% of PDF/FinePDFs-DE candidates reach exact five-shingle Jaccard similarity 0.8, so candidate coverage is not verified near-duplicate prevalence.Across six directional comparisons, 19% to 67% of fuzzy-only candidates reach 0.8; the nominal LSH setting favors recall but imposes no hard threshold.Source: §F.1 · p. 21, Figure 11 · p. 22
• Candidate coverage varies with document length: HTML/FineWeb2-DE peaks at 17% for 1,000–10,000 characters, while PDF/FinePDFs-DE rises from 7% below 1,000 characters to 24% at 100,000 or more.Thus, an aggregate candidate rate does not describe overlap uniformly across lengths.Source: §F.1 · p. 21, Figure 11 · p. 22

F.2 Reference-Corpus Intersections

• Format-matched-only memberships make up 99.3% of the HTML and 96.8% of the PDF normalized-exact unions, while candidate memberships also overlap across reference corpora.151,207 HTML documents and 54,616 PDFs are candidates in both FineWeb2-DE and FinePDFs-DE but not German Commons; LSH membership does not verify a fixed similarity threshold.Source: §F.2 · p. 22, Figure 12 · p. 23

G Rehydration Diagnostics

• Missing resources, robots.txt exclusions, and DNS failures account for 25%, 21%, and 13% of HTML records, and 22%, 19%, and 14% of PDF records.Together, these categories explain 91% of unsuccessful HTML downloads and 93% of unsuccessful PDF downloads; weighted availability reaches 75% for HTML and 83% for PDF in 2025.Source: G Rehydration Diagnostics · p. 23, Figure 13 · p. 25
• After standardizing by crawl-year distribution, the Top 10% minus Other 90% download gap is 6.3 percentage points for HTML and 7.6 for PDF.The unadjusted gaps are 10.1 and 11.4 points, respectively; source recency explains part, but not all, of the association between quality rank and availability.Source: G Rehydration Diagnostics · p. 23, Figure 13 · p. 25
• Among successfully extracted records, median normalized five-shingle similarity is 0.809 for Top 10% versus 0.278 for Other 90% HTML, and 0.982 versus 0.939 for PDF.These conditional similarities describe fidelity among successful extractions and should not be combined directly with unconditional rates.Source: G Rehydration Diagnostics · p. 24

H Quality-Score Construction and Diagnostics

• The 0–132 quality score is a convenience ranking that double-weights content quality, educational value, reasoning indicators, and commercial bias.It excludes language, academic sector, technical content, audience level, content safety, and PII annotations; commercial bias is reverse-coded.Source: H Quality-Score Construction and… · p. 25, Table 6 · p. 26