Source-linked AI summary
Latin BERT: A Contextual Language Model for Classical Philology
David Bamman, Patrick J. Burns
TL;DR
Latin BERT addresses the absence of a contextual model for a historical language by training on 642.7 million tokens spanning 22 centuries. It achieves state-of-the-art POS tagging on all three Latin Universal Dependency datasets and supports further Latin NLP and scholarly applications.
Problem
The introduction of a contextual language model for Latin addresses the lack of such a model for a historical language.
Source: §5 · p. 9Method
Results
Latin BERT reaches new state-of-the-art POS tagging on all three Latin Universal Dependency datasets and achieves 75.4% word-sense-disambiguation accuracy versus 67.3% for static embeddings.
Source: §1 · p. 2, §4.3 · p. 7Takeaways & Limitations
The model supports Latin NLP tasks and offers contextual-neighbor search and word-probability estimates for textual emendation.
Source: §5 · p. 9The word-sense-disambiguation evaluation uses citation examples longer than five words, and the authors note that it could be expanded with more training and evaluation examples.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
We present Latin BERT, a contextual language model for the Latin language, trained on 642.7 million words from a variety of sources spanning the Classical era to the 21st century. In a series of case studies, we illustrate the affordances of this language-specific model both for work in natural language processing for Latin and in using computational methods for traditional scholarship: we show that Latin BERT achieves a new state of the art for part-of-speech tagging on all three Universal Dependency datasets for Latin and can be used for predicting missing text (including critical emendations); we create a new dataset for assessing word sense disambiguation for Latin and demonstrate that Latin BERT outperforms static word embeddings; and we show that it can be used for semantically-informed search by querying contextual nearest neighbors. We publicly release trained models to help drive future work in this space.
1 Introduction
Contextual language models have advanced NLP across tasks, while language-specific models have improved several tasks over multilingual baselines. This work brings that approach to Latin and presents a model, evaluation resources, and scholarly applications.
1 Introduction
- Contextual language models have advanced tasks including parsing, word-sense disambiguation, and coreference, motivating language-specific models beyond English and multilingual BERT.Prior language-specific models reported improvements on several NLP tasks over an mBERT baseline.Source: §1 · p. 1
- Latin research has developed corpora, treebanks, and NLP tools that have supported literary-critical work on intertextuality and stylometry.The cited examples include research on text reuse, genre, and authorship.Source: §1 · p. 1
- The authors train and openly release a Latin BERT model on 642.7 million tokens from sources spanning the Classical era to the present.The model extends the growing set of language-specific BERT models described in the introduction.Source: §1 · p. 2
- Latin BERT establishes new state-of-the-art POS tagging performance on all three Universal Dependency datasets for Latin.This is the paper's reported benchmark result.Source: §1 · p. 2
- The authors create a Latin word-sense-disambiguation dataset from Lewis and Short dictionary examples.The dataset uses the dictionary’s sense inventory and attestations.Source: §1 · p. 2
- The paper illustrates Latin BERT in four case studies, including text infilling and contextual-neighbor search.The cases span NLP and digital Classics applications.Source: §1 · p. 2
2 Corpus
The corpus combines Latin texts from diverse sources and periods, balancing noisier Internet Archive material with other subcorpora. Latin-specific tokenization and WordPiece segmentation prepare the data for BERT.
2 Corpus
- The corpus draws on sources spanning Classical texts to contemporary Latin, with Internet Archive scans filtered to retain books meeting a 40% vocabulary-coverage threshold.The other sources include the Latin Library, Patrologia Latina, Corpus Thomisticum, and Latin Wikipedia; non-Internet Archive texts are upsampled to balance training at approximately 50% per group.Source: §2 · p. 2
- Texts are segmented with Latin-specific CLTK sentence and word tokenizers, including separation of enclitics from their host words.A Latin-specific WordPiece vocabulary of 32,895 subword types is learned; representations for multi-subtoken words are averaged.Source: §2 · pp. 2–3
- WordPiece segmentation represents subword units rather than the full space of inflected word forms, reducing the representation space for highly inflected Latin.For example, audentes is split into two subtokens, whose representations are averaged in experiments.Source: §2 · p. 3
3 Training
Latin BERT is a 12-layer model with 768-dimensional hidden representations, pretrained using whole-word masking for one million steps. Training took about five days on a TPU v2.
3 Training
- The model has 12 layers and a hidden dimensionality of 768.The maximum sequence length was 256 WordPiece tokens, with a batch size of 256.Source: §3 · p. 3
- The model is pretrained with whole-word masking for one million steps on a TPU.Training took approximately five days on a TPU v2.Source: §3 · p. 3
- Training cost $540 on Google Cloud, and the resulting model was converted for use with HuggingFace.The trained model and supporting code were made available online.Source: §3 · p. 3
4 Analyses
The analyses test Latin BERT across POS tagging, textual emendation, word-sense disambiguation, and contextual search. Results include state-of-the-art POS tagging on three Latin dependency datasets and evidence that contextual representations capture useful distinctions and similarities.
4.1 POS tagging
- For 200 sampled instances, Latin BERT’s representations nearly separate cum used as ADP from cum used as SCONJ without POS-specific training.The analysis uses final-layer token representations and t-SNE; several apparent exceptions reflect collocations or annotation issues.Source: §4.1 · p. 3, Figure 1 · p. 4
- Latin BERT establishes a new state of the art for POS tagging on all three evaluated Latin datasets, with a 4.6-point improvement on Perseus.The comparison covers the Perseus, Index Thomisticus, and PROIEL treebanks.Source: §4.1 · p. 4, Table 2 · p. 5
4.2 Text infilling
- On 2,205 emendations, Latin BERT reconstructs the human-judged word in 33.1% of cases, ranks it in the top 10 in 62.2%, and in the top 50 in 74.0%.The evaluation uses bracketed conjectures from Latin texts and excludes examples whose centered 5-gram appears in training data; one case shows the model ranking secundo first while human emendation primo also ranks highly.Source: §4.2 · pp. 5–6, Table 3 · p. 5
4.3 Word Sense Disambiguation
- The new Lewis and Short-based WSD dataset contains 8,354 examples across 201 headwords, with balanced binary labels for each headword’s first two major senses.Examples are drawn from dictionary attestations, with sentence-length and minimum-example criteria applied before splitting the data by headword.Source: §4.3 · pp. 6–7
- Latin BERT reaches 75.4% accuracy on the WSD dataset, an 8.1-point absolute improvement over static embeddings at 67.3%.The dataset is balanced, so random choice yields 50% accuracy.Source: §4.3 · p. 7, Table 5 · p. 7
- The WSD evaluation uses dictionary citations longer than five words, and the authors note that retrieving full-text examples could expand the available data.The dictionary examples are often fragmentary; 55% are fewer than five words long.Source: §4.3 · pp. 6–7
4.4 Contextual Nearest Neighbors
- Contextual nearest-neighbor queries retrieve tokens similar in usage, capturing both the accusative, division-related sense of in and semantic or phrase variants of audentes.The study represents 16 million tokens and finds token-level comparisons more precise than sentence-level comparisons using [CLS].Source: §4.4 · pp. 7–8, Table 6 · p. 8, Table 7 · p. 8
5 Conclusion
The paper presents the first contextual language model for a historical language, training Latin BERT on 642.7 million tokens spanning 22 centuries. Its case studies show applications to Latin NLP and computational scholarship, while the authors identify further tasks and representation analyses for future work.
5 Conclusion
- Latin BERT is the first contextual language model for a historical language, trained on 642.7 million tokens written from 200 BCE to the present.Source: §5 · p. 9
- Case studies apply the model to POS tagging and word sense disambiguation, and explore word probabilities for textual emendation and contextual nearest neighbors for semantic similarity.Source: §5 · p. 9
- Future work could probe which BERT layers suit specific tasks, extend evaluation to parsing and named entity recognition, and explore further applications in Classics.The reported work uses representations from the single final layer; the authors also point to lexicography and other literary-critical applications.Source: §5 · p. 9
Appendix
Table 8 presents Latin BERT training hyperparameters.
Appendix
- Table 8 presents Latin BERT training hyperparameters.Source: Table 8 · p. 14