Source-linked AI summary
Spanish Pre-trained BERT Model and Evaluation Data
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, Jorge Pérez
TL;DR
Spanish language-model training and evaluation resources are difficult to find despite Spanish’s wide use. The paper addresses this gap with a Spanish-only BERT model and GLUES, a Spanish GLUE-style benchmark. Spanish BERT generally outperforms multilingual BERT-based models across tasks and achieves new state-of-the-art results on some datasets.
Problem
Spanish NLP lacks readily available resources for training and evaluating language models.
Method
The paper pre-trains a BERT-based model exclusively on Spanish data and compiles Spanish tasks into the GLUES benchmark.
Results
Spanish BERT outperforms most reported multilingual BERT results and achieves new state-of-the-art results on POS and MLDoc.
Takeaways & Limitations
The released model, corpus, and benchmarks provide free resources for Spanish NLP research and applications.
Takeaways & Limitations
The Spanish-only models cannot use English or other-language training data, and larger multilingual XLM-RoBERTa models outperform them on reported Spanish XNLI and NER results.
Abstract
from arXiv · showhide
The Spanish language is one of the top 5 spoken languages in the world. Nevertheless, finding resources to train or evaluate Spanish language models is not an easy task. In this paper we help bridge this gap by presenting a BERT-based language model pre-trained exclusively on Spanish data. As a second contribution, we also compiled several tasks specifically for the Spanish language in a single repository much in the spirit of the GLUE benchmark. By fine-tuning our pre-trained Spanish model, we obtain better results compared to other BERT-based models pre-trained on multilingual corpora for most of the tasks, even achieving a new state-of-the-art on some of them. We have publicly released our model, the pre-training data, and the compilation of the Spanish benchmarks.
1 INTRODUCTION
Spanish NLP lacks accessible resources for training and evaluation despite the language’s broad use. The paper introduces a Spanish-only BERT model, a GLUE-style benchmark, and publicly released resources.
- Spanish NLP research faces limited access to resources for training and evaluating language models.
- The paper presents the first BERT model pre-trained specifically for Spanish.
- GLUES compiles several Spanish-specific tasks in one repository following the spirit of the GLUE benchmark.
- Fine-tuning Spanish BERT produces better results than multilingual BERT-based models on most tasks and reaches new state-of-the-art results on some.
- The authors release the model, pre-training corpus, and benchmark compilation as free resources.
2 RELATED WORK
Pre-trained language models combine large-scale language-model training with supervised task adaptation, while BERT uses self-attention and masked language modeling. Prior multilingual and monolingual models establish the context for Spanish BERT.
- ULMFiT pre-trains on vast text with language modeling, then fine-tunes supervisedly for specific tasks.
- BERT replaces recurrent architecture with self-attention and reconstructs tokens corrupted through masked language modeling.
- Multilingual BERT models learn from corpora in multiple languages and provide strong baselines for non-English tasks.
- Single-language BERT models often outperform multilingual models, motivating a Spanish-specific counterpart and evaluation.
3 SPANISH-BERT MODEL, DATA AND TRAINING
Spanish BERT is a BERT-Base-sized model trained in cased and uncased versions on a large Spanish corpus with subword vocabularies and enhanced masking strategies.
- The model has 12 self-attention layers, 12 heads per layer, hidden size 768, and 110M parameters.
- The authors train cased and uncased versions using Spanish data from Wikipedia and OPUS sources.
- The training corpus contains about 3 billion words and is comparable in size to the original BERT corpus.
- Each model uses a 32K-token vocabulary built from 31K SentencePiece subwords plus 1K placeholder tokens.
- Training incorporates 10x Dynamic Masking and Whole-Word Masking techniques.
4 GLUES
GLUES is a Spanish-language benchmark that assembles common NLP tasks for standardized evaluation. It includes inference, paraphrase, question answering, tagging, classification, and parsing datasets.
- GLUES compiles common Spanish NLP tasks following the original English GLUE benchmark.
- Natural Language Inference: XNLI evaluates three-class natural language inference—entailment, contradiction, or neutral—using accuracy.
- Paraphrasing: PAWS-X tests whether two Spanish sentences are semantically equivalent and is evaluated with accuracy.
- Additional tasks: GLUES also includes CoNLL named entity recognition, Universal Dependencies POS tagging, MLDoc classification, and dependency parsing.
- Question Answering: Question answering uses MLQA, TAR, and XQuAD translated datasets and reports exact match and F1 scores.
5 EVALUATION
The evaluation compares Spanish BERT with multilingual BERT under Spanish-only fine-tuning, using task-specific output layers and standard fine-tuning procedures. Spanish BERT outperforms most multilingual results, while remaining disadvantaged where multilingual or translated-data settings matter.
- 5.1 FINE-TUNING: Fine-tuning uses WordPiece tokenization, Adam optimization, weight decay, learning-rate warmup, and linear decay.The learning rate is warmed up for the first 10% of steps before linear decay.
- 5.1 FINE-TUNING: Spanish BERT is compared with the best multilingual BERT results using Spanish training data for each dataset.The evaluation follows a setup designed for comparison with multilingual BERT models and uses task-specific output layers.
- 5.1 FINE-TUNING: Sentence length is limited to 128 tokens, with sliding windows retaining the last 64 tokens for word-level classification tasks.Each subsequent window receives 64 retained tokens and 64 new tokens.
- 5.1 FINE-TUNING: The hyperparameter search varies batch size {16, 32}, learning rate {5e-5, 3e-5, 2e-5}, and epochs {2, 3, 4}.The authors state that an extensive hyperparameter search remains for future work.
- 5.2 RESULTS: Spanish BERT outperforms most multilingual BERT results and reaches new state-of-the-art results on POS and MLDoc, except in some question-answering settings.The largest difference appears on XNLI; nearly half of MLQA’s 81K examples have answer-position mismatches, possibly reflecting translation quality issues.
- 5.2 RESULTS: Spanish-only models cannot use English or other-language training data, whereas multilingual models can; XLM-RoBERTa and mBERT exceed reported Spanish-only results in specified tasks.XLM-RoBERTa reports 85.6% for XNLI and 89% for NER, while mBERT reports 90.7% on Spanish PAWS-X when using the English training set.
6 CONCLUSION
The paper releases a Spanish-only pretrained model, its training corpus, and Spanish evaluation benchmarks. It presents these resources as a way to broaden access to Spanish NLP research and applications, while identifying model efficiency as a future direction.
- 6 CONCLUSION: The authors pretrain and open-source a Spanish-only model together with its training corpus and evaluation benchmarks.The paper frames these releases as community resources.
- 6 CONCLUSION: The released Spanish-BERT model is intended to encourage research and deep-learning applications in Spanish-speaking countries.The authors connect pretrained-model ease of use with broader downstream use by practitioners outside computer science.
- 6 CONCLUSION: Future work targets Spanish NLP models with lower size, memory, and computation requirements, including ALBERT models ranging from 5M to 223M parameters.The authors report encouraging initial results with smaller models and plan to release them.