Source-linked AI summary
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
Pedro Javier Ortiz Suárez, Laurent Romary, Benoît Sagot
TL;DR
Mid-resource languages often lack contextualized-embedding data with sufficient volume and genre diversity, while Common Crawl offers scale at the cost of noise. The paper trains monolingual ELMo models from filtered OSCAR and Wikipedia data for five languages and evaluates them on POS tagging and parsing. OSCAR embeddings consistently outperform Wikipedia-trained ELMo and match or improve state-of-the-art results across the five languages.
Problem
Wikipedia may be too small and genre-limited for high-quality contextualized embeddings in some languages, motivating evaluation of the quantity–quality trade-off.
Method
The paper trains monolingual ELMo embeddings from Wikipedia and filtered Common-Crawl-based OSCAR data for five languages, evaluating them on POS tagging and dependency parsing.
Results
OSCAR-based ELMo consistently outperforms Wikipedia-based ELMo and equals or improves state-of-the-art tagging and parsing results across all five languages.
Takeaways & Limitations
Filtered Common-Crawl data can train high-quality contextualized embeddings for languages lacking sufficient textual volume or genre variety.
Takeaways & Limitations
The study does not replicate experiments with state-of-the-art fine-tuning models because their training costs exceed the available computational infrastructure.
Abstract
from arXiv · showhide
We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.
1 Introduction
Transfer learning and contextualized representations have advanced neural NLP, but contextualized models require large corpora to address polysemy and broad usage. This paper tests whether larger, noisier OSCAR data can outperform smaller, narrower Wikipedia data for five languages.
- Transfer learning methods use pretrained representations either as features for task-specific architectures or through fine-tuning.
- Contextualized embeddings generate word representations from surrounding context, allowing the same word to receive different representations.
- ELMo and Flair provide feature-based contextualized representations, while BERT and related transformers commonly use fine-tuning.
- Contextualized language models can train on raw text, but they require enormous quantities of data to represent polysemy and diverse language use.
- Wikipedia offers multilingual coverage but may be too small for some languages and is limited to one genre, whereas Common Crawl increases quantity and style coverage while introducing noise.
- The paper trains OSCAR-based and Wikipedia-based ELMo embeddings for five languages and evaluates them on POS tagging and dependency parsing.
2 Related work
Prior work established multilingual and monolingual contextual representations, but comparable high-quality monolingual resources remained limited. Related parsing and tagging systems increasingly combined contextual embeddings with neural architectures and multilingual training.
- Common Crawl improved fixed word-embedding performance, especially for mid- to low-resource languages, while Wikipedia supported multilingual embeddings across many languages.
- mBERT provided a single Wikipedia-trained multilingual model for 100 languages, alongside a small set of distributed monolingual ELMo models.
- No comparable large-scale effort had yet produced high-quality monolingual contextualized representations across many non-English languages.
- The CoNLL 2018 parsing and tagging leaders coupled neural architectures with ELMo embeddings, including a training cap of 20 million words.
- UDPipe 2.0 with mBERT and UDify extended multilingual contextual modeling, with UDify training one model on 124 treebanks across 75 languages.
3 Corpora
The study compares Wikipedia and Common-Crawl-derived OSCAR corpora for five typologically diverse languages, measuring corpus size and noise before training contextualized embeddings. OSCAR uses language classification, character-based filtering, and deduplication, while retaining the scale and diversity of web text.
- Corpora: The experiments train Wikipedia-only and OSCAR-only ELMo embeddings for Bulgarian, Catalan, Danish, Finnish, and Indonesian.
- Wikipedia: Wikipedia corpora were extracted from XML dumps, tokenized with UDPipe, and retained without deduplication because they were already small.
- OSCAR: Common Crawl is a freely available web repository whose multilingual snapshots contain noise and are not classified by language.
- OSCAR: OSCAR derives from a November 2018 Common Crawl snapshot exceeding 20 terabytes and applies language classification, filtering, and deduplication.
- OSCAR: OSCAR filtering keeps lines with at least 100 UTF-8-encoded characters, correcting the disparity created by a 100-byte threshold between ASCII and non-ASCII languages.
- OSCAR: OSCAR was selected because it remained the only very large, easily downloadable Common-Crawl-based corpus, despite newer CCNet filtering not being available.
- Noisiness: Noise was estimated by testing dictionary membership for one-million-word random samples from each language's OSCAR and Wikipedia corpora.
4 Experimental Setting
The experiments compare Wikipedia- and OSCAR-trained ELMo models across five languages, attaching each to UDPipe 2.0 for POS tagging and dependency parsing. Models are trained and evaluated under controlled settings across six UD treebanks.
- Experimental design: The study varies the ELMo model while keeping UDPipe 2.0 hyperparameters at default values, apart from the number of training tokens.UDPipe 2.0 uses gold tokenization and segmentation for each training run.
- Contextualized word embeddings: ELMo models are trained for Bulgarian, Catalan, Danish, Finnish and Indonesian on both OSCAR and Wikipedia corpora.Each model is trained for 10 epochs, with checkpoints saved after epochs 1, 3 and 5 to examine possible overfitting on smaller Wikipedia corpora.
- UDPipe 2.0: UDPipe 2.0 jointly predicts POS tags, lemmas and dependency trees using shared and task-specific recurrent layers.Tagging and lemmatization use bidirectional LSTMs with Softmax classifiers, while parsing uses a parser-specific bidirectional LSTM followed by biaffine attention.
- UDPipe 2.0: Pretrained, trainable, character-level and contextualized representations are concatenated before entering UDPipe 2.0’s shared bidirectional LSTM layers.The contextualized ELMo representations are added to the three embeddings in the original implementation.
- Evaluation data: Evaluation uses all available treebanks for the five languages in UD version 2.2, covering six treebanks with varied textual domains.The treebanks include legal, news, fiction, oral, grammatical, web, blog, parliamentary and other texts.
5 Results & Discussion
Across five languages, OSCAR-trained ELMo embeddings consistently outperform Wikipedia-trained ELMo embeddings on POS tagging and dependency parsing, while often exceeding multilingual and previous state-of-the-art systems. Results also examine training duration and computational cost, including a Finnish-specific performance plateau and the difficulty of comparing epochs across differently sized corpora.
- Parsing and POS tagging results: OSCAR-based ELMo models achieve state-of-the-art results for the reported tagging and parsing metrics across the five languages.The comparisons include UDPipe 2.0, UDPipe 2.0+mBERT, and UDify.
- Parsing and POS tagging results: OSCAR-based ELMo embeddings consistently outperform Wikipedia-based ELMo embeddings across POS tagging and dependency parsing in all five languages.The evaluation uses UDPipe 2.0 and reports UPOS, UAS, and LAS against baseline and state-of-the-art systems.
- Parsing and POS tagging results: 300MB Danish Wikipedia data improves the UDPipe 2.0 baseline without showing overfitting, and all Wikipedia models avoid a negative impact when added to the baseline.The Danish result is presented as a closer examination of the smaller-corpus behavior.
- Parsing and POS tagging results: Finnish ELMoWikipedia exceeds previous state-of-the-art results on all metrics for Finnish-FTB and on UPOS for Finnish-TDT.For Finnish, multilingual baselines can fall below the UDPipe 2.0 baseline, whereas the monolingual OSCAR model improves previous state-of-the-art results by more than 2 points for some metrics.
- Impact of the number of training epochs: Wikipedia ELMo results generally improve through 10 epochs, with the fully trained models consistently giving the best results, while OSCAR ELMo shows the same pattern except for Finnish.Finnish OSCAR performance caps around epochs 3–5 despite its corpus being more than 20 times larger than the smallest Wikipedia corpus.
- Impact of the number of training epochs: Ten epochs correspond to substantially different training-step counts across corpora, such as 78K steps for Indonesian Wikipedia versus 2.6M for Indonesian OSCAR.The authors therefore argue that training steps may provide a more transparent comparison than epochs.
- Computational cost and carbon footprint: Training-cost analysis reports power draw, training time, energy consumption, and CO2 emissions for each ELMo model, but not for UDPipe 2.0.UDPipe models took less than four hours to train, and shared-machine usage made their power consumption difficult to estimate accurately.
- Computational cost and carbon footprint: Replicating the experiments with transformer fine-tuning models was beyond the available computational infrastructure because those architectures are extremely costly to train.The authors frame ELMo as retaining a useful performance–training-cost trade-off.
6 Conclusions
The paper concludes that properly filtered OSCAR data supports high-quality contextualized embeddings for five mid-resource languages, outperforming Wikipedia-based embeddings across POS tagging and dependency parsing.
- OSCAR-based ELMo embeddings consistently outperform Wikipedia-based embeddings on POS tagging and dependency parsing.
- Properly filtered Common Crawl data is not massively noisier than Wikipedia.
- OSCAR-based embeddings improve state-of-the-art results for POS tagging and dependency parsing across all six selected treebanks.
- Common-Crawl-based OSCAR can train high-quality contextualized embeddings where standard resources lack sufficient volume or genre variety.
A.1 Number of training steps for each checkpoint and each corpus
Table 8 reports the number of training steps for each checkpoint of the ELMoWikipedia and ELMoOSCAR models for each language.
- Table 8 organizes training-step counts by checkpoint for ELMoWikipedia and ELMoOSCAR across languages.
- The table supports comparison of training progress between the Wikipedia and OSCAR ELMo models.