Source-linked AI summary
German's Next Language Model
Branden Chan, Stefan Schweter, Timo Möller
TL;DR
The paper investigates how training data, model size, and Whole Word Masking affect German BERT and ELECTRA models for downstream NLP tasks. It presents GBERT and GELECTRA models that achieve state-of-the-art results, including a 4.3% improvement on GermEval14 NER over the previous best.
Problem
The paper asks whether varying training data, model size, and Whole Word Masking can produce strong German language models for classification and NER tasks.
Method
The authors train seven German BERT and ELECTRA models using different data combinations, model sizes, and Whole Word Masking configurations, then evaluate them on downstream tasks.
Results
The models set state-of-the-art results across GermEval18 and GermEval14; GELECTRALarge improves over the previous best by +4.3% on GermEval14.
Takeaways & Limitations
The experiments provide practical guidance on training regimes and model configurations for teams building German language models under compute constraints.
Takeaways & Limitations
Because most training data comes from internet-scraped OSCAR text, the models may inherit explicit content, unverified information, and social biases despite filtering.
Abstract
from arXiv · showhide
In this work we present the experiments which lead to the creation of our BERT and ELECTRA based German language models, GBERT and GELECTRA. By varying the input training data, model size, and the presence of Whole Word Masking (WWM) we were able to attain SoTA performance across a set of document classification and named entity recognition (NER) tasks for both models of base and large size. We adopt an evaluation driven approach in training these models and our results indicate that both adding more data and utilizing WWM improve model performance. By benchmarking against existing German models, we show that these models are the best German models to date. Our trained models will be made publicly available to the research community.
1 Introduction
The paper presents German BERT and ELECTRA models and uses downstream evaluation to select effective checkpoints under limited compute. It also examines how training data and Whole Word Masking affect performance.
- The authors introduce German GBERT and GELECTRA models for classification and NER tasks.The models are based on BERT and ELECTRA architectures.
- GELECTRALarge improves state-of-the-art GermEval18 performance by about +4% on coarse and +2.5% on fine hate-speech detection.
- GELECTRALarge also reaches state-of-the-art on GermEval14 NER, outperforming the previous best by over +4%.
- Increasing training data by an order of magnitude and using Whole Word Masking are evaluated as sources of performance gains.
- Checkpoints are regularly evaluated on classification and NER tasks, with the best-performing checkpoint selected as final.This downstream-oriented approach replaces saving only after a fixed number of training steps.
2 Related work
The related work traces a shift from recurrent contextual language models to Transformer-based BERT and ELECTRA systems. It also motivates Whole Word Masking and highlights multilingual modeling and dataset-bias concerns.
- Earlier contextual language models such as ELMo, ULMFiT, and FLAIR use LSTM-based architectures.These models established benchmarks on text classification, PoS tagging, and NER.
- More recent language models use Transformer architectures, including GPT-2, BERT, RoBERTa, ALBERT, and ELECTRA.
- BERT pretraining masks about 15% of input tokens and learns primarily from those masked positions.
- ELECTRA replaces selected tokens synthetically and trains a discriminator to detect original versus substituted tokens at every input position.A jointly trained generator supplies plausible replacements.
- Whole Word Masking masks all subword tokens belonging to a word and is reported to improve the training signal and task performance.
- German models include monolingual systems from DBMDZ and deepset alongside multilingual BERT and XLM-RoBERTa.
- Naturally occurring training data can reproduce cultural biases, motivating critical evaluation and better dataset documentation.
3 Datasets
The study combines several German corpora for pretraining, including web, Wikipedia, OPUS, and legal text. The authors document both dataset sizes and risks from predominantly web-scraped data.
- German OSCAR contributes 145GB of monolingual Common Crawl text, while the German Wikipedia corpus contributes 6GB.OSCAR is language-filtered and preprocessed Common Crawl data.
- OPUS contributes around 10GB from domains including movie subtitles, parliament speeches, and books.
- The dataset overview reports corpus sizes in gigabytes.
- Open Legal Data contributes about 2.4GB of German court decisions.
- Because OSCAR supplies 89% of training data from internet-scraped text, filtering cannot guarantee removal of explicit material, misinformation, or social biases.The authors caution against uncritical deployment in settings requiring impartiality.
- GermEval18 provides coarse and fine hate-speech classification tasks with two and four classes, respectively.
- GermEval14 contains over 31,000 German sentences and 590,000 tokens with nested NER annotations across 12 classes.
4 Training
The authors train seven German BERT and ELECTRA variants while varying model size, data, and Whole Word Masking. Models are matched to standard base or large architectures and released in selected forms.
- Model configurations: Seven models vary combinations of training data, model size, and Whole Word Masking for BERT.
- Model configurations: GBERT variants use the same cased vocabulary as DBMDZ BERTBase and match BERTBase or BERTLarge size.
- BERT variants: GBERTData uses all available data without Whole Word Masking.
- BERT variants: GBERTWWM uses DBMDZ BERTBase data with Whole Word Masking, while GBERTData + WWM combines all data with Whole Word Masking.
- BERT variants: GBERTLarge uses all available data, Whole Word Masking, and the large model configuration.
- ELECTRA variants: GELECTRA variants match ELECTRABase or ELECTRALarge size and differ primarily in training data.
- ELECTRA variants: GELECTRA uses DBMDZ BERT data, GELECTRAData uses all available data, and GELECTRALarge uses all available data at large size.
- Release: The best models are uploaded to the Hugging Face model hub as four deepset GBERT and GELECTRA checkpoints.
5 Evaluation
The evaluation uses continuous checkpointing during pretraining and downstream testing across two classification tasks and one NER task. Classification checkpoints are evaluated repeatedly, while the best-performing checkpoint is selected for each model.
- Evaluation procedure: Checkpoints are regularly saved, converted with Hugging Face Transformers, and evaluated on GermEval18 Coarse, GermEval18 Fine, and GermEval14.The first two are hate-speech classification tasks; GermEval14 is a NER task.
- Evaluation procedure: Each checkpoint is evaluated three times on both document-classification tasks because the authors observed substantial run-to-run variance.NER checkpoints are evaluated once without early stopping.
- Checkpoint selection: For every model, the reported result uses the checkpoint with the best downstream performance.Classification runs use early stopping and different random seeds.
- Baselines: The evaluation pipeline also benchmarks the trained models against German BERT and multilingual mBERTBase and XLM-RoBERTaLarge models.The German baselines are deepset German BERTBase and DBMDZ German BERTBase.
6 Results
The models make most of their downstream gains early in pretraining, while later improvements are more incremental. Across the benchmark tasks, the trained models improve substantially over prior results, and extra data and Whole Word Masking generally help.
- Training dynamics: Most downstream gains occur during the first phase of pretraining, with later improvements becoming more incremental.The best checkpoint occurs at different training points for different models.
- Benchmark results: +3.93% is GELECTRALarge’s improvement over the previous SoTA on GermEval18 (Coarse).GBERTData + WWM, XLM-RoBERTaLarge, and GBERTLarge also improve upon the previous SoTA.
- Benchmark results: +2.45% is GELECTRALarge’s improvement over the previous SoTA on GermEval18 (Fine).XLM-RoBERTaLarge exceeds the previous best by +1.39% on this task.
- Ablations: Extra data gives modest gains, with GBERTData + WWM outperforming GBERTWWM by +0.93% and GELECTRAData outperforming GELECTRA by +1.59%.GBERTData nevertheless performs -0.2% below DBMDZ BERTBase.
- Ablations: Whole Word Masking consistently improves the BERT models, with gains of +1.25% over DBMDZ BERTBase and +2.38% over GBERTData.These comparisons are reported for GBERTWWM and GBERTData + WWM, respectively.
7 Discussion
Larger models perform substantially better, while training curves suggest several models remain undertrained and could improve with continued training. GELECTRA also outperforms GBERT despite seeing fewer tokens, although model-size comparisons are confounded by differing training regimes.
- 7.1 Model Size: +5.31% averaged F1 separates GELECTRALarge from GELECTRAData, while GBERTLarge exceeds GBERTData + WWM by +2.33%.The authors note that large models also received many more training tokens than their base counterparts.
- 7.2 Training Length: GELECTRA and GELECTRAData show clear upward training trends, suggesting that both models are undertrained.GBERTWWM and GBERTData + WWM also trend upward during the second half of training.
- 7.2 Training Length: None of the models shows clear overfitting or performance degradation, so further training may improve performance.The conclusion is based on the downstream evaluation graphs in Figure 1.
- 7.3 ELECTRA Efficiency: GELECTRALarge outperforms GBERTLarge by +1.47% averaged F1 after seeing only half as many tokens.The comparison follows 1 million training steps and is presented as evidence for ELECTRA's learning efficiency.
- 7.4 Instabilities: A performance dip around 2 million steps coincides with stopping, saving, and reloading the base GBERT models, but its exact cause was not investigated.The authors suspect the events are related but leave the mechanism outside the project's scope.
8 Conclusion
The authors trained German GBERT and GELECTRA models across different training regimes and architectures. Their evaluations provide practical reference points for compute-limited teams and report new state-of-the-art performance on GermEval18 and GermEval14.
- 8 Conclusion: The trained German models vary in both training regime and model architecture.This variation forms the basis for the reported comparisons.
- 8 Conclusion: The experiments offer compute-limited teams guidance on batch sizes and training lengths for efficient language-model training.The authors frame these results as data points for practitioners training models from scratch.
- 8 Conclusion: The evaluated GELECTRA and GBERT models set new state-of-the-art performance for both large and base models on GermEval18 and GermEval14.The claim covers both model families and both model sizes.