BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling

1 Introduction

• Training corpora for large language models grew from 40GB to nearly 570GB, while many relevant models first appeared in English and later in less-performing multilingual versions.The introduction notes that resources comparable to English are not always available in other languages, and that monolingual models can take years to appear.Source: §1 · p. 2
• The paper asks how much data is enough for a well-performing monolingual model, how to select documents when data are abundant, and how data quality affects training.These questions motivate the study of sampling from web-crawled data.Source: §1 · p. 2
• The study tests whether sampling can reduce training-data size and time without noticeably affecting final model performance, using Spanish and RoBERTa.The authors identify Spanish as their test language and explore sampling documents from a large web-crawled dataset.Source: §1 · p. 2

2.1 Spanish mC4

• Spanish mC4 contains natural text from Common Crawl and is part of the multilingual mC4 corpus used to train mT5.The Spanish portion contains about 416 million documents and 235 billion words.Source: §2.1 · p. 2
• Sampling from Spanish mC4 produced subsets of about 50M documents and 200GB, enabling training with roughly one fifth of the data and half the steps of regular RoBERTa-base pre-training.The full Spanish corpus contains about 416M documents and approximately 1TB of uncompressed data.Source: §2.2 · p. 2, §2.1 · p. 2

2.2 Perplexity sampling

• Unlike filtering, perplexity sampling retains documents while using statistical quartiles to bias against potentially poor-quality text.The authors distinguish their sampling approach from CCNet's equal-sized perplexity splits and note that higher-perplexity portions there achieved slightly better results.Source: §2.2 · p. 3
• The authors calculate document perplexity with a 5-gram KenLM model trained on Spanish Wikipedia, using raw, unnormalized lines without SentencePiece tokenization.They chose to skip SentencePiece for speed; the method's perplexity values may therefore reflect writing styles closer to Wikipedia.Source: §2.2 · p. 3
• Stepwise sampling oversamples the central perplexity quartiles and subsamples the rest, while Gaussian weighting smooths the Stepwise scheme's sharp boundaries.The Stepwise method aims to bias against very low-perplexity short or repetitive texts and very high-perplexity potentially poor-quality texts; its α parameter was adjusted to about 10% of Q3.Source: §2.2 · p. 3
• The sampling schemes were adjusted to draw about 50M documents from 416M, reducing raw data from 1TB to 200GB; random sampling served as the baseline.The sampled validation split was too small, so the authors drew 50k evaluation documents from training data and excluded them from training.Source: §2.2 · p. 4

3 Training

• Training followed Liu et al. (2019) hyperparameters for 250,000 steps, moving from sequence length 128 to 512 on a TPUv3-8.The first 230,000 steps used length 128; batch sizes were 2048 at length 128 and 384 at length 512.Source: §3 · p. 4
• For the length-512 stage, random sampling retained optimizer state, while Stepwise and Gaussian sampling initialized new optimizers.The paper notes that clear details for increasing sequence length during training were unavailable.Source: §3 · p. 4
• Training one model for each sampling method took roughly a week on the reported hardware.The hardware was three TPUv3-8 devices.Source: §3 · p. 5

4 Evaluation

• Evaluation covered CoNLL 2002 NER and POS, PAWS-X paraphrase identification, and XNLI natural language inference.Comparisons included mBERT, BETO, and the Barcelona Supercomputing Center's RoBERTa model.Source: §4 · p. 5
• All models were fine-tuned for five epochs with maximum sequence length 512 and batch size 16.Fine-tuning used learning rate 5e-5 on two NVIDIA Quadro RTX6000 GPUs.Source: §4 · p. 5
• Gaussian-sampled BERTIN models outperformed the BETO and BNE baselines on NER and PAWS-X, while BERTIN performed well overall.The evaluation summarized results across all tasks; the supplied passage does not give task-level scores.Source: §4 · p. 5

5 Bias and ethics

• The authors frame bias assessment as necessary because the publicly available models could be used in real-world applications with serious effects on people.They describe their own bias analysis as basic and point to the importance of making model limitations clear.Source: §5 · p. 5

5.1 Bias examples

• Even a quick, shallow analysis found apparent bias, alongside some predictions that were more neutral than anticipated.Examples included “son” as the first completion for “do the dishes” and no “pink” among color recommendations for a girl.Source: §5.1 · p. 6
• Gender examples also place women at “home” or “the bed,” while associating men with public places such as streets and cities.The paper contrasts these completions with some less stereotyped examples, including “son” for “do the dishes.”Source: §5.1 · p. 6
• Race and religion prompts produced mostly neutral or repetitive suggestions but also troubling associations, including “racist” as a suggestion for Jews.Chinese people received both “intelligent” and “stupid”; the paper notes this can hint at different forms of racism.Source: §5.1 · p. 6
• The model favored Spain-common vocabulary over Latin American alternatives, including “coche” over “carro” and “conducir” over “manejar.”For “take the bus,” it chose “coger,” which the paper notes is inappropriate in most Latin American countries.Source: §5.1 · p. 6

6 Discussion

• The models achieved state of the art on MLDoc and were virtually tied on UD-POS, with masked-language accuracy between 0.65 and 0.69.Source: §6 · p. 7
• Gaussian sampling led in four of seven tasks, while generally outperforming other sampling methods except on POS.Stepwise also scored better than random at a similar number of steps; the authors say more statistical analysis is needed.Source: §6 · p. 7
• Random sampling lost performance during sequence-length extension, whereas Gaussian-512 outperformed Gaussian-128 on the main and downstream tasks.The authors suggest optimizer handling may relate to the difference and say a new optimizer is a safe alternative when uncertain or compute is scarce.Source: §6 · p. 7

7 Further Work

• Replicating the training on a large model was not possible within roughly 10 days of access to 3 TPUv3-8.Source: §7 · p. 7
• Further experiments should examine Gaussian parameters and whether Wikipedia-trained perplexity sampling biases selected text toward Wikipedia-like language.The authors note that colloquial and informal social-media language may be underrepresented.Source: §7 · p. 7
• The authors propose exploring whether perplexity sampling can be combined with deduplication, which they describe as having a complementary philosophy.Source: §7 · p. 7

8 Conclusions

• With roughly 10 days of access to 3 TPUv3-8, the models surpassed prior state of the art on a few tasks and improved document classification over models trained with far more resources.Source: §8 · p. 7
• The large available datasets posed a challenge in constrained environments, while subsampling showed promise for improving access for smaller teams.The authors hope other groups will use and improve these techniques.Source: §8 · pp. 7–8
• Reduced training time and data could support iterations on newer datasets designed to address bias, including training RoBERTa from scratch on such datasets.Source: §8 · p. 8

Appendix: Mask Prediction

• The appendix covers masked-token predictions in settings involving gender, religion, race, origin, and geographic bias.Source: Appendix: Mask Prediction · p. 10
• Each masked token is shown with five predictions in descending probability order.Source: Appendix: Mask Prediction · p. 10
• Translations of prediction terms are provided whenever possible.Source: Appendix: Mask Prediction · p. 10

• The dishes prompt asks who should be told to do them and lists “hijo,” “madre,” “jefe,” “pareja,” and “suegra.”Source: Gender · p. 10
• For “Las mujeres conducen muy 〈mask〉,” the listed completions include “alto,” “rápido,” “poco,” “fuerte,” and “bien.”Source: Gender · p. 10
• The prompts place women at “casa” and men at “calle” among the proposed completions for where each belongs.Source: Gender · p. 10
• The paired “tenía que ser mujer/hombre” prompts offer different proposed completions, including “hacer” for women and “hecho” for men.Source: Gender · p. 10
• For “Como soy chica, mi color favorito es el 〈mask〉,” the listed colours are “rojo,” “blanco,” “azul,” “verde,” and “naranja.”Source: Gender · p. 10

Religion

• For the religion prompts, the listed completions include group-name repetitions, “católicos” for Christians, and “racistas” among the suggestions for Jews.The accompanying discussion says the first suggestion tends to repeat the group and describes other suggestions as generally neutral and tame, while noting the Jewish and Chinese examples.Source: Religion · pp. 10–11

Race and origin

• Race and origin prompts list “inteligentes” and “tontos” for Chinese people, alongside completions such as “asiáticos” and “negros.”The discussion notes that the Chinese examples include both “intelligent” and “stupid” suggestions.Source: Race and origin · p. 11

Geography

• The geography examples show regional vocabulary differences, including “coche” in Spain and “carro” in Latin America, and note that “coger” has different meanings across regions.Source: Geography · p. 11

Appendix: Availability

• The appendix says the source code is available under Apache License 2.0 and that the sampled dataset’s train split has 50M samples, with validation taken from original mC4.It also describes on-the-fly sampling from mC4 and lists the language model and fine-tuned versions as available.Source: Appendix: Availability · p. 11