PaperlayerView exact revision on arXiv

Source-linked AI summary

MarIA: Spanish Language Models

Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies, Joan Llop-Palao, Joaquín Silveira-Ocampo, Casimiro Pio Carrino, Aitor Gonzalez-Agirre, Carme Armentano-Oller, Carlos Rodriguez-Penagos, Marta Villegas

arXiv:2107.07253v5cs.CLcs.AI

TL;DR

Spanish lacked large language models trained with vast, high-quality data. MarIA presents Spanish models and associated resources, and reports better overall generalization than existing models.

  • Problem

    Spanish lacked large language models trained with vast, high-quality data, while most massive pretrained models were trained for English.

    Source: §1 · p. 1

  • Method

    The paper presents and publishes Spanish RoBERTa and GPT2 models alongside a pretraining corpus and a new Question Answering dataset.

    Source: §3.2.1 · p. 5, §6 · p. 8

  • Results

    MarIA models are reported to generalize better overall than existing models.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The project aims to make large-scale Spanish models, high-quality corpora, and evaluation sets available to industry and the scientific community.

    Source: §1 · p. 1

    PAWS-X contains machine-translated text, so some Spanish sentences may not be entirely correct.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

This work presents MarIA, a family of Spanish language models and associated resources made available to the industry and the research community. Currently, MarIA includes RoBERTa-base, RoBERTa-large, GPT2 and GPT2-large Spanish language models, which can arguably be presented as the largest and most proficient language models in Spanish. The models were pretrained using a massive corpus of 570GB of clean and deduplicated texts with 135 billion words extracted from the Spanish Web Archive crawled by the National Library of Spain between 2009 and 2019. We assessed the performance of the models with nine existing evaluation datasets and with a novel extractive Question Answering dataset created ex novo. Overall, MarIA models outperform the existing Spanish models across a variety of NLU tasks and training settings.

1 Introduction

MarIA addresses the lack of large Spanish models trained on high-quality data by presenting models and resources for industry and research. The authors report better overall generalization than existing models.

  • MarIA aims to provide industry and the scientific community with large-scale Spanish models, high-quality corpora, and evaluation sets.Source: §1 · p. 1
  • Across comparisons with existing models, MarIA models are reported to generalize better overall.Source: §1 · pp. 1–2
  • The project processed the largest clean Spanish corpus to date from National Library of Spain web crawls conducted between 2009 and 2019.Source: §1 · p. 2
  • The project trained RoBERTa-base, RoBERTa-large, GPT2, and GPT2-large models and created the new Spanish Question Answering dataset SQAC.Source: §1 · p. 2
  • The models were evaluated on a diverse set of tasks.Source: §1 · p. 2

2 Related Work

Related work traces language modeling from word embeddings to larger Transformer-based models, alongside work on Spanish datasets, representations, and language-specific models.

  • Language modeling began with next-word prediction, producing fixed vector representations known as word embeddings.Source: §2 · p. 2
  • Spanish research has built datasets and computed word representations using language-modeling algorithms.Source: §2 · p. 2
  • Language modeling later scaled to larger datasets and expressive models, including LSTM-based models and Transformer-based BERT and GPT families.The cited examples include Peters et al. (2018), Vaswani et al. (2017), Devlin et al. (2018), and Radford et al. (2018; 2019a).Source: §2 · p. 2
  • Spanish language-specific models included BETO, trained on existing corpora including OPUS and Spanish Wikipedia, and BERTIN, trained on Spanish mC4.BERTIN is cited to Xue et al. (2020).Source: §2 · p. 2
  • For Spanish, the authors chose RoBERTa for encoder models and GPT2 for decoder models.RoBERTa is cited to Liu et al. (2019), and GPT2 to Radford et al. (2019a).Source: §2 · p. 2

3 Data

The section describes a BNE web corpus cleaned for pretraining and an evaluation workbench, including SQAC, a newly created Spanish extractive QA dataset. Its data choices address the limited size, variety, and Spanish-originality of existing QA resources.

3.1 Pretraining corpus

  • The corpus pipeline parsed and repaired text, filtered documents and sentences, applied language identification, removed duplicates, and preserved document boundaries.Sentence-level filtering used a cascade of language identifiers; deduplication used Onion’s 5-gram overlap method.Source: §3.1 · p. 3
  • 570GB of clean, deduplicated data—over 200M documents and 135B tokens—remained from BNE crawls conducted between 2009 and 2019.The source corpus excludes Wikipedia; its release awaits the BNE’s legal determination.Source: §3.1 · pp. 3–4

3.2 Fine-tuning datasets

  • The nine-task evaluation workbench includes textual entailment, document classification, named-entity recognition, paraphrase identification, POS tagging, semantic similarity, and QA.The section also describes CoNLL-NERC and CAPITEL-NERC datasets.Source: §3.2 · p. 4
  • Existing Spanish QA resources lacked a sizable training set; XQuAD was evaluation-only, while translation-based data could be insufficiently varied or contain translation artifacts.The authors describe XQuAD as professionally translated but not large or varied enough for proper training or evaluation.Source: §3.2 · pp. 4–5

3.2.1 SQAC

  • For nearly 600 randomly selected questions, answer-span annotation received 85% F1 and 71% EM after normalization.SQAC’s guidelines sought context-answerable questions using varied interrogatives and encouraged synonyms to limit lexical overlap.Source: §3.2.1 · p. 5
  • SQAC adds Spanish-original material to benchmarks that often rely on translations and are relatively small and limited in genre or topic variety.The dataset is publicly available on HuggingFace.Source: §3.2.1 · p. 5

4 Language Models

MarIA comprises RoBERTa encoder models and GPT2 generative models, trained with their respective language-modeling objectives. The section specifies four model sizes and notes a sequence-length constraint for generative models.

4 Language Models

  • RoBERTa uses masked language modeling without next-sentence prediction, while GPT2 uses next-token prediction.Both architectures use byte-level BPE.Source: §4 · pp. 5–6
  • The four configurations range from RoBERTa-base at 125M parameters to GPT2-large at 774M parameters.RoBERTa-large has 355M parameters and GPT2 has 117M parameters.Source: §4 · p. 5
  • Generative models were trained with a 512-token sequence length because of computational constraints; the authors suggest a sliding window for longer tasks.Pretraining used one epoch without dropout and retained weight decay of 0.01.Source: §4 · p. 6

5 Evaluation

The evaluation compares MarIA models with multilingual and Spanish baselines across NLP tasks, and reports smooth GPT2 perplexity convergence. RoBERTa-large leads most tasks, with its clearest differences on non-Wikipedia datasets.

5 Evaluation

  • GPT2 and GPT2-large perplexity curves converge smoothly on training and validation data, while GPT2-large requires significantly more updates.The paper presents these curves because a lack of evaluation datasets prevented a proper GPT2 benchmark.Source: §5 · p. 6, Figure 1 · p. 7, Figure 2 · p. 7

5.1 Baselines

  • Baselines include multilingual mBERT and Spanish BETO, BERTIN, and ELECTRICIDAD.The comparison covers nine tasks; GPT2 models are instead assessed with training and validation perplexity curves because evaluation datasets are lacking.Source: §5.1 · p. 6

5.2 Fine-tuning methodology

  • Classification tasks add one linear layer and use the BERT [CLS] or RoBERTa <s> token, with a maximum input length of 512 tokens.Table 3 reports train, development, and test set sizes for each downstream task.Source: §5.2 · pp. 6, 8, Table 3 · p. 8
  • Models used the same HuggingFace fine-tuning defaults and a grid search, with the best development-set configuration evaluated on the test set.The search covered batch sizes 16 and 32, weight decay 0.01 and 0.1, learning rates 1e-5, 3e-5, and 5e-5, and up to five epochs.Source: §5.2 · p. 6, §5.3 · p. 8

5.3 Results

  • RoBERTa-large leads most tasks, while RoBERTa-base wins some; non-Wikipedia tasks show 2-point CAPITEL-NERC and almost 3-point STS and SQAC differences, possibly reflecting Wikipedia-data contamination.For MLDoc, differences are marginal and BETO slightly surpasses the other models.Source: §5.3 · p. 8

6 Conclusions

MarIA combines Spanish pretraining data, an extractive QA dataset, and pretrained models; the authors present these resources as a step toward reducing the gap with English and other high-resource languages.

  • The work introduces a pretraining corpus, a new Spanish QA dataset, and large pretrained language models.Source: §6 · p. 8
  • The pretraining corpus is described as more diverse than previous Spanish datasets such as Wikipedia, drawing on myriad sources.Source: §6 · p. 8
  • SQAC is presented as a significant, high-quality contribution that allows appropriate evaluation of Spanish extractive QA systems.Source: §6 · p. 8
  • The two RoBERTa models show high performance on many downstream NLP tasks, alongside two generative GPT2 models of different sizes.Source: §6 · p. 8
  • The paper concludes that its contributions are a crucial step toward reducing the NLP gap with English and other high-resource languages.Source: §6 · p. 8
  • Future work includes extending the corpus with sources such as Wikipedia or books, analyzing its topics and bias, and scaling model context and size.The authors also aim to improve inference efficiency; the stated context-length target is 1024 rather than 512.Source: §6 · p. 8

Appendix I

The appendix tables identify best evaluation configurations and their metrics for several datasets, but the supplied captions do not provide score values.

Appendix II

Appendix II gives sample masked-language-model prediction assessments, pairing Spanish prompts with candidate completions from several models.

  • The appendix presents a sample of masked-language-model prediction assessments, with prompts followed by model-specific candidate completions.Source: Appendix II · p. 15
  • Some prompts concern gendered contexts, including women's role in science and what matters to her or him, followed by model-specific suggestions.Source: Appendix II · p. 16
  • Other prompts include statements about who may wear skirts, violence, and danger in the street, with candidate completions from multiple models.Source: Appendix II · pp. 15, 17
  • Additional examples prompt completions about a problem's solution, a required decision, and what Felipe thinks about Juan and Marta.Source: Appendix II · p. 19

Appendix III

The appendix describes publicly available FastText embeddings as an additional resource, not an evaluated component of the models.

  • The authors computed 300-dimensional FastText word embeddings using both CBOW and Skip-gram, and made them available on Zenodo.Source: Appendix III · p. 21
  • Processing the document-level clean data took around 20 days on an HPC node with an AMD EPYC 7742 processor and 128 threads.Source: Appendix III · p. 21
  • The embeddings are provided as an additional resource and are not evaluated in the paper.Source: Appendix III · p. 21

Appendix IV

Appendix IV presents text-completion examples for GPT2 and GPT2-large. The examples show each model continuing several Spanish prompts.

Appendix IV

  • GPT2 and GPT2-large are both illustrated with Spanish text-completion examples across several prompts.The appendix labels separate sets of examples for the two models.Source: Appendix IV · p. 21
  • GPT2 completes prompts about Barcelona, the Barcelona Supercomputing Center, and the pandemic.Examples include a Barcelona destination and a continuation about saving lives during the pandemic.Source: Appendix IV · p. 21
  • GPT2-large examples include continuations about scientific research, vaccination, and university volunteer work.One continuation suggests scientific production and research as possible uses for a Spanish GPT2 model.Source: Appendix IV · p. 21
Loading 2107.07253v5…