Source-linked AI summary
The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models
Lukas Gienapp, Christopher Schröder, Stefan Schweter, Christopher Akiki, Ferdinand Schlatt, Arden Zimmermann, Phillipe Genêt, Martin Potthast
TL;DR
Large-scale, verifiably licensed German training data is scarce, while web-scraped alternatives can carry licensing issues. The German Commons aggregates openly licensed text and yields 154.56 billion tokens, the largest collection of open German text to date.
Problem
Method
Results
154.56 billion tokens of openly licensed German text make the German Commons the largest collection of open German text to date.
Source: §7 · p. 8Takeaways & Limitations
Explicit canonical SPDX license URLs on every document enable unambiguous downstream use, supporting ethically compliant German-language model development.
Source: §7 · p. 882.27% of tokens come from news and cultural domains, leaving scientific (0.54%) and economic (0.07%) content critically underrepresented.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Large language model development relies on large-scale training corpora, yet most contain data of unclear licensing status, limiting the development of truly open models. This problem is exacerbated for non-English languages, where openly licensed text remains critically scarce. We introduce the German Commons, the largest collection of openly licensed German text to date. It compiles data from 41 sources across seven domains, encompassing legal, scientific, cultural, political, news, economic, and web text. Through systematic sourcing from established data providers with verifiable licensing, it yields 154.56 billion tokens of high-quality text for language model training. Our processing pipeline implements comprehensive quality filtering, deduplication, and text formatting fixes, ensuring consistent quality across heterogeneous text sources. All domain subsets feature licenses of at least CC-BY-SA 4.0 or equivalent, ensuring legal compliance for model training and redistribution. The German Commons therefore addresses the critical gap in openly licensed German pretraining data, and enables the development of truly open German language models. We also release code for corpus construction and data filtering tailored to German language text, rendering the German Commons fully reproducible and extensible.
1 Introduction
The German Commons addresses the scarcity of verifiably openly licensed German pretraining data with a large corpus and a reproducible processing library. The paper also analyzes the corpus and includes a datasheet.
1 Introduction
- Verifiably open training texts matter because opaque data can undermine trust, while sensitive or copyrighted text may be reproduced by generative models.The paper notes risks including personally identifiable information and uncertainty about benchmark contamination or copyrighted material in web crawls.Source: §1 · p. 1
- 154.56 billion tokens across 35.78 million documents and seven thematic domains make the German Commons a pretraining-scale collection of openly licensed German text.Table 1 overviews document counts, token counts, and median sequence lengths by source domain.Source: §1 · p. 1, Table 1 · p. 1
- The authors release the llmdata processing library for reproducibility, analyze corpus properties, and include a datasheet following Gebru et al..Source: §1 · p. 1
2 Related Work
The related work contrasts the scale of web-scraped corpora with their licensing, privacy, and quality risks, and describes German and openly licensed alternatives as fragmented or difficult to verify. The German Commons is positioned as a unified source of verifiably licensed, high-quality text.
Web Corpora for LLM Training
- Web-scraped corpora such as C4 [5], The Pile [7], and FineWeb [13] [14] draw almost exclusively on Common Crawl, bringing licensing, privacy, and quality risks alongside scale.Substring license matching does not establish verifiable license scope; terms of service may prohibit model training, PII persists despite preprocessing, and heterogeneous web data requires extensive filtering.Source: §2 · p. 2
German Text Datasets
Openly Licensed Training Corpora
- Natural-language licenses are harder to verify than machine-readable code licenses, so open-text efforts turn to trusted government, GLAM, and Wikimedia sources.Questionable content has appeared in large openly licensed corpora [1], while code repositories can provide explicit licensing annotations [22].Source: §2 · p. 2
- Existing collections include the 228B-token Open License Corpus [23], KL3M’s 1.35 trillion English tokens [16], Common Pile’s roughly two trillion English tokens [24], and Common Corpus’s 2 trillion multilingual tokens [25].KL3M excludes share-alike licenses such as CC-BY-SA, ruling out Wikipedia; Common Corpus includes 112 billion German tokens and serves as a starting point for the German Commons.Source: §2 · p. 2
3 Sourcing Open German Text Data
The German Commons combines existing and newly assembled German text under explicit licensing and quality criteria, with provenance documented across seven thematic domains. Its sourcing approach prioritizes traceable rights and institutional providers while drawing on both published corpora and extracted texts.
3 Sourcing Open German Text Data
- Stricter licensing and quality criteria reduce usable tokens from the German subset of Common Corpus from 112B to 70B, before expansion with updated versions and new collections.The collection uses the newest available versions through August 31, 2025; Table 2 describes constituent datasets and their source, license, type, and size.Source: §3 · p. 2, Table 2 · p. 3
3.1 Where do we obtain data from?
- Sources include established corpora with published texts and metadata collections requiring extraction, gathered from institutional catalogs and trusted Zenodo and Hugging Face uploads.Where source texts are unavailable, the authors crawl and extract plain text; non-curated-platform uploads are limited to trusted parties with clear licensing protocols.Source: §3.1 · p. 2
3.2 What do we consider openly licensed?
- Every document must have an explicit license; the authors exclude ambiguous content rights and licenses limiting commercial use, redistribution, or modification.Accepted licenses follow the Open Knowledge Foundation’s Open Definition 2.13, include public-domain-equivalent, attribution, and copyleft categories, and are recorded using SPDX-canonical identifiers; users are advised to check license compatibility.Source: §3.2 · pp. 2, 4
- The authors rely on providers’ license statements rather than independent audits, acknowledging misattribution risk and limiting inclusion to established institutions and verified platforms.They describe this as a viable large-scale approach given the scarcity of open German text, while applying less strict criteria than Kandpal et al. [24].Source: §3.2 · p. 4
3.3 Detailed Provenance Information
- The seven domains are Web, Political, Legal, News, Economics, Cultural, and Scientific, with Legal separated from Political because the authors consider its writing style distinct.The domains cover collaborative and user-generated text; parliamentary and judicial materials; newspapers; business and trade; books and speech; and scholarly and educational content.Source: §3.3 · p. 4
- Web text includes collaborative platforms, discussions, comments, video transcripts, and licensed GitHub files; Political and Legal collections draw on parliamentary records, court decisions, and laws.The web sources include Wikipedia and Wikivoyage texts, Wikipedia discussions, Austrian newspaper comments, and YouTube transcripts; legal materials include German court proceedings and European laws.Source: §3.3 · p. 4
- News sources include German newspaper editions from 1671–1994, around 1,600 Austrian newspapers, and more than 4 million Europeana documents; Economics includes EU public procurement notices.The news collections also include public-domain releases and community-written Wikinews articles; procurement notices come from the online EU Official Journal Supplement.Source: §3.3 · p. 5
- Cultural sources span historical literature, public-domain collections, digitized books, and speech transcripts; Scientific sources include periodicals, textbooks, open-access books, licensed arXiv articles, and extracted OpenAlex PDFs.The scientific collection keeps only the latest version of each arXiv article to reduce deduplication overhead.Source: §3.3 · p. 5
4 Data Processing
The section presents an ordered processing pipeline for making heterogeneous inputs consistent, applying quality filters, and fixing text formatting. The resulting corpus is distributed as Parquet files partitioned by thematic domain and source dataset.
Data Processing
- The ordered pipeline produces a consistent dataset schema and Parquet files partitioned by domain and source for selective loading.Appendix A.4 reports documents and tokens removed at each processing step and by source dataset.Source: §4 · p. 5
Plain Text Extraction
Text Formatting
- Text formatting applies FTFY fixes and custom regex transformations to correct encoding, character, and OCR-related whitespace artifacts.The transformations normalize Unicode and quotes, collapse excess spaces and tabs, preserve paragraph boundaries, remove line-break hyphenation, and trim line whitespace.Source: §4 · p. 5
Language & Length Filtering
- FastText [57] language identification discards texts with less than 0.65 probability of being primarily German, and sequences under 32 GPT-2 tokens are removed.The compressed 176-language model [58] classifies snippets truncated to 4096 characters; token counts are stored with the text.Source: §4 · p. 5
Quality Filtering
- Quality filters use percentile thresholds on language-filtered, formatted data and add OCR-specific checks for casing anomalies, fragmented words, and special-character density.Indicators include word and symbol statistics, alphabetic and stop-word ratios, and line, paragraph, character, and n-gram repetition; documents outside the 5th or 95th percentile are removed as appropriate, with thresholds manually checked by cursory inspection.Source: §4 · pp. 5–6
Deduplication
- Paragraph-level deduplication uses Dolma’s [11] LSH bloom filter with 20-gram MinHash shingles and an 0.8 collision rate.Paragraphs are split at newlines, all but one colliding chunk are removed, and the bloom filter uses a 1e-4 false-positive rate; its file is released for deduplicating new data against the corpus.Source: §4 · p. 6
PII Removal
- Regex filters and Presidio remove PII while replacing detected information with generic text to preserve sentence structure.The removed types include email addresses, phone numbers, IP addresses, credit-card numbers, IBAN numbers, and URLs.Source: §4 · p. 6
License Mapping and Filtering
- License identifiers are mapped to canonical SPDX license URLs, and documents are retained only when their licenses are open and all licenses listed for them are permissive.Source: §4 · p. 6
5 Corpus Statistics
The section characterizes German Commons by context-length and license distributions, filtering retention, and classifier-assessed text properties. It reports that every thematic domain appears across context-length partitions and that sampled text has minimal toxicity and varied linguistic complexity and sentiment.
Token and Document Distribution By Thematic Domain
- At short contexts, web content supplies 43.31% of tokens, while news supplies 78.60% at medium and 63.38% at long contexts.Cultural content supplies 83.73% of very-long-context tokens; all domains remain represented in every partition, enabling exposure control through sub- or oversampling.Source: §5 · pp. 6–7
Token and Document Distribution By License Type
- 126.61B tokens (74.291% of the corpus) are public-domain-equivalent, compared with 34.48B attribution tokens (20.40%) and 7.93B copyleft tokens (4.69%).Public-domain-equivalent licenses represent 35.96% of documents; attribution and copyleft represent 33.4% and 30.60%, respectively. Public-domain tokens concentrate in cultural (39.36%) and news (57.39%) text; attribution tokens in web content (88.76%); and copyleft tokens in web (52.29%) and political (33.84%) text.Source: §5 · p. 7, Table 6 · p. 7, Table 7 · p. 7
Filtering Statistics
- 46.41% of initial data was removed by quality filtering, 2.7% more by deduplication, and 0.08% by license-compliance and PII filtering.Overall retention was 50.73%, although most source corpora retained 70%–95%; quality filtering mainly removed non-German and very short text, while duplicates were concentrated in web and news corpora.Source: §5 · p. 7
Text Properties
- 0 sampled paragraphs scored above 3 in any toxicity category, and 95% scored 0 on average.The toxicity assessment covered 385,467 paragraphs sampled across sources; scores 0–3 were considered non-toxic, with negligible differences between domains.Source: §5 · p. 7, Table 8 · p. 7
- 65% of sampled text was classified as everyday language and 30% as special language, with 2% plain and 3% easy.Special language was highest in scientific content (63.8%), while everyday language was highest in web content (81.4%); cultural text was nearly evenly divided between everyday (52.2%) and special language (39.9%).Source: §5 · p. 7, Figure 2 · p. 8
- 80.5% of sampled text was neutral, compared with 16.4% negative and 3.1% positive.Web content had the highest negative share (31.8%), cultural content the highest positive share (5.0%), and news the highest neutral share (92.0%).Source: §5 · p. 8, Figure 3 · p. 8
6 Limitations and Ethical Considerations
The German Commons inherits limitations from its sources and curation, including temporal, OCR, linguistic, demographic, cultural, and privacy risks. Documentation and released tools support informed use and future extensions.
Limitations and mitigation
- News (47.02%) and cultural content (35.25%) make up 82.27% of tokens, creating historical and nostalgia bias, while scientific (0.54%) and economic (0.07%) domains are underrepresented.The authors identify adding contemporary German text as a priority for rebalancing the temporal distribution.Source: §6 · p. 8
- OCR-heavy sourcing risks residual errors, especially in German diacritics [67]; OCR filtering is applied, but LLM correction was omitted over cost, hallucination, and historical-text misinterpretation risks.Specialized correction models could improve the corpus in future.Source: §6 · p. 8
- Standard German and formal institutional registers dominate, with likely socioeconomic and Western Protestant cultural biases; openly licensed dialectal and minority varieties are a proposed remedy.The authors note that targeted inclusion depends on these varieties becoming available under open licenses.Source: §6 · p. 8
- Regex- and Presidio-based PII removal provides only surface-level privacy protection, though historical skew and public-record sources may reduce potential adverse effects.Residual PII may still be present in the data.Source: §6 · p. 8
Ethical considerations and documentation
- Documentation following established frameworks, provenance and license metadata, bloom filters, thematic partitions, and a scalable preprocessing pipeline support informed use and future contributions.Bloom filters enable cross-corpus contamination detection, while thematic partitioning supports selective use.Source: §6 · p. 8
7 Conclusion
The German Commons addresses the scarcity of large-scale, verifiably licensed German training data with 154.56 billion tokens of openly licensed text from institutional sources. It combines seven thematic domains with systematic quality controls and explicit license metadata.
7 Conclusion
- 154.56 billion tokens make the German Commons the largest collection of open German text to date, addressing scarce verifiably licensed training data and licensing issues in web-scraped alternatives.The collection aggregates openly licensed text from institutional sources to enable German language model development.Source: §7 · p. 8
- The corpus spans seven domains—web, political, legal, news, economics, cultural, and scientific—and applies quality filtering, deduplication, and PII removal.Corpus statistics show the data's suitability for model training and verify the high quality of the provided text.Source: §7 · p. 8
- Every document carries an explicit canonical SPDX license URL, enabling unambiguous downstream use and supporting ethically compliant German language model development.Source: §7 · p. 8
A Data Filtering · A.1 Quality Filtering Parameter Values
The quality filter excludes documents that cross thresholds for text form, duplication, character and spacing anomalies, or unusual word-length patterns. Its parameter values specify exclusion cutoffs for each measure.
A.1 Quality Filtering Parameter Values
- Text-form exclusion rules cover alphabetic words, bullets, ellipses, hashes, and stop-word counts.Documents are excluded when the Alphabetic Word Ratio is > 0.99, Bullet Line Ratio > 0.70, Ellipsis Line Ratio > 0.30, Ellipsis Ratio > 0.10, Hash Ratio > 0.10, or Stop-word Count < 6.00.Source: §A.1 · p. 11
- Document-level repetition cutoffs apply to duplicated lines and paragraphs, measured by line or character fractions.Exclusion thresholds are > 0.25 for Duplicated Line Fraction, > 0.15 for Duplicated Lines Character Fraction, > 0.30 for Duplicated Paragraph Fraction, and > 0.20 for Duplicated Paragraph Character Fraction.Source: §A.1 · p. 11
- Character concentration in the most frequent n-grams is limited by separate top-2-, top-3-, and top-4-gram cutoffs.The Top-2-Gram Character Fraction threshold is > 0.07, Top-3-Gram Character Fraction > 0.10, and Top-4-Gram Character Fraction > 0.13; the measure is the character share accounted for by the most frequent n-gram.Source: §A.1 · p. 11
- Anomaly thresholds target spacing, capitalization, OCR fragments and artifacts, unusual Unicode, repeated patterns, and numbers embedded in words.Exclusion cutoffs are > 0.15 for Spacing Anomaly Ratio, > 0.10 for Case Anomaly Ratio, > 0.20 for Word Fragment Ratio, > 0.25 for Line Artifact Ratio, > 0.03 for Special Character Density, > 0.05 for Repeated Character Ratio, and > 0.08 for Numeric Context Errors.Source: §A.1 · p. 11
- Word-length filters exclude texts with unusually short or long average words, low or high variation, or excessive very short or very long words.Cutoffs are Avg. Word Length (min) < 4.80 and (max) > 7.30; Word Length Standard Deviation (min) < 1.00 and (max) > 5.00; Very Short Words Ratio > 0.10 and Very Long Words Ratio > 0.10.Source: §A.1 · p. 11
A.2 Stopwords
The section lists German stopwords grouped across articles, conjunctions, verbs, prepositions, pronouns, adverbs, contractions, question words, quantifiers, and particles.
A.2 Stopwords
- Listed stopwords include definite and indefinite articles, conjunctions, common verbs, and prepositions.Definite articles: der, die, das, den, dem, des; indefinite articles: ein, eine, einen, einem, einer; conjunctions: und, oder, aber; common verbs: ist, sind, hat, haben, wird, werden; prepositions: von, zu, mit, in, auf, für, bei, nach, vor, über, unter, durch, gegen, ohne, um.Source: §A.2 · p. 11
- The list also includes pronouns, adverbs, subordinating conjunctions, and contractions.Pronouns: ich, du, er, sie, es, wir, ihr, sich, sein, seine, ihrer, ihren, mich, dich; adverbs: nicht, auch, nur, noch, schon, hier, dort, da, dann, jetzt, heute, sehr, mehr, weniger, ganz, gar, etwa; subordinating conjunctions: dass, wenn, als, wie; contractions: an, am, im, ins, zum, zur, vom, beim.Source: §A.2 · p. 11
- Additional listed groups are question words, quantifiers, modal verbs, and particles.Question words: was, wer, wo, wann, warum, wie, welche, welcher; quantifiers: alle, viele, einige, andere, jede, jeden, jeder; modal verbs: kann, könnte, muss, soll, will, würde; particles: ja, nein, doch, so, also, nun, mal.Source: §A.2 · p. 11
A.3 PII Generic Replacement Values
The section presents generic replacement values for PII, organized by category, replacement, and comment.
A.3 PII Generic Replacement Values
- The PII replacement table is structured around category, generic replacement, and comment.The supplied passage contains the table headers but no replacement entries.Source: §A.3 · p. 12
A.4 Processing Step Statistics … Ethical Considerations
The German Commons is a large, curated collection of openly licensed German text, assembled through documented processing and distribution practices. The datasheet describes its composition, intended uses, licensing, maintenance, and ethical limitations.
A.4 Processing Step Statistics
- The processing-statistics table tracks document and token counts through initial, filtered, deduplicated, and final stages, with percentages showing the share remaining.Initial counts follow metadata filtering where available; final counts include PII replacement and final license filtering.Source: §A.4 · p. 12
Collection Process
- Collection used direct downloads, API crawling, and automated document extraction, followed by quality filtering, deduplication, and validation against source materials.All qualifying content was retained after deterministic selection for German language, explicit open licensing, quality, and institutional verification; collection used versions available through August 31, 2025.Source: Datasheet: German Commons · p. 13, Collection Process · p. 13
Datasheet: German Commons
- The datasheet reports that content came directly from original sources, with metadata drawn from repository records and extracted text checked against source documents.Authors used automated systems without crowdworkers or external annotators; the underlying content spans multiple centuries, predating and extending beyond collection in 2025.Source: Datasheet: German Commons · p. 13
Motivation
- The collection addresses the shortage of large-scale open German text and supports pretraining, broader German NLP research, domain-specific development, and cross-domain evaluation.Compilation was supported by German and European research grants, ScaDS.AI, and OpenWeb-Search.eu; constituent datasets primarily come from state-funded institutions in Germany and Austria.Source: Motivation · p. 13
Dataset Composition
- 35,778,211 documents comprise 154,558,196,961 GPT-2 tokens, with document metadata and no supervised labels.Instances include source, license, domain, token-count, perplexity, and OCR-quality metadata. Paragraph deduplication may alter source texts; residual cross-corpus duplicates and extraction artifacts may remain.Source: Dataset Composition · p. 13
Data Preprocessing
- The dataset is freely available without fees or registration, while each document retains its original license and users must follow its terms.The compilation metadata is licensed under ODC-BY 1.0; commercial use is permitted by all constituent licenses, which may require attribution or share-alike provisions.Source: Data Preprocessing · p. 13
Dataset Maintenance
- Authors maintain the dataset and may update it as new collections emerge; the open-source pipeline supports reviewed community extensions and reproducible processing.The preprocessed dataset is distributed as Parquet files through public repositories, with Hugging Face Hub as the primary distribution location.Source: Dataset Maintenance · p. 13
Ethical Considerations
- No formal ethics review was conducted; the dataset uses publicly available, explicitly licensed material, removes PII, and may contain content offensive by contemporary standards.Its historical and temporal breadth makes comprehensive content moderation infeasible, and documents may include information relating to individuals.Source: Ethical Considerations · p. 13