Source-linked AI summary
Probing Pretrained Language Models for Lexical Semantics
Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, Anna Korhonen
TL;DR
The paper addresses the limited systematic evidence on lexical type-level knowledge in pretrained language models. It evaluates extraction strategies across six languages and five lexical tasks, finding both universal best practices and language- and task-specific variation, with lexical information concentrated in lower but distributed across multiple layers.
Problem
Systematic evidence was lacking on how pretrained language models encode lexical type-level knowledge across languages, tasks, and representation-extraction choices.
Method
The study evaluates type-level embeddings extracted from monolingual and multilingual LMs across six languages and five lexical tasks, comparing extraction configurations and fastText.
Results
Monolingual LMs contain more lexical information than multilingual models, special tokens hurt performance, and lower-layer averaging outperforms single-layer or higher-layer alternatives.
Takeaways & Limitations
Recommended choices are monolingual LMs, multiple contexts, excluded special tokens, and averaging over lower layers; BERT-based embeddings can recover substantial lexical knowledge.
Takeaways & Limitations
The discrepancy between monolingual and cross-lingual task results requires further investigation, including whether linear maps are suboptimal for BERT-based word vectors.
Abstract
from arXiv · showhide
The success of large pretrained language models (LMs) such as BERT and RoBERTa has sparked interest in probing their representations, in order to unveil what types of knowledge they implicitly capture. While prior research focused on morphosyntactic, semantic, and world knowledge, it remains unclear to which extent LMs also derive lexical type-level knowledge from words in context. In this work, we present a systematic empirical analysis across six typologically diverse languages and five different lexical tasks, addressing the following questions: 1) How do different lexical knowledge extraction strategies (monolingual versus multilingual source LM, out-of-context versus in-context encoding, inclusion of special tokens, and layer-wise averaging) impact performance? How consistent are the observed effects across tasks and languages? 2) Is lexical knowledge stored in few parameters, or is it scattered throughout the network? 3) How do these representations fare against traditional static word vectors in lexical tasks? 4) Does the lexical information emerging from independently trained monolingual LMs display latent similarities? Our main results indicate patterns and best practices that hold universally, but also point to prominent variations across languages and tasks. Moreover, we validate the claim that lower Transformer layers carry more type-level lexical knowledge, but also show that this knowledge is distributed across multiple layers.
1 Introduction and Motivation
The paper investigates how pretrained language models encode lexical semantic knowledge and presents a systematic analysis across languages, tasks, and extraction configurations.
- Prior probing research examined syntactic, morphological, semantic, and world knowledge, but systematic multilingual evidence on lexical semantic information remained limited.
- The study spans six typologically diverse languages and five lexical tasks to evaluate type-level lexical representations extracted from pretrained LMs.
- Extraction configurations vary by source LM, context, special tokens, subword selection, external corpora, and the Transformer layers averaged.
- The central questions concern cross-language and cross-task generalisation, parameter and layer concentration, competitiveness with fastText, and similarities across monolingual LM representations.
- Different languages and tasks require distinct configurations, while lexical information is concentrated mainly in lower layers and benefits from averaging multiple layers.
2 Lexical Representations from Pretrained Language Models
The paper converts contextual Transformer representations into static type-level word embeddings and studies how extraction choices affect their lexical information.
- Transformer representations differ from classical static vectors because they depend on token context and position, and may represent subword strings.
- Prior work reported context-sensitive word-sense information in higher layers, motivating analysis of where abstract type-level lexical information is encoded.
- The study uses unsupervised word-level extraction strategies rather than learned probes to construct representations for lexico-semantic tasks.
- The extraction pipeline compares monolingual and multilingual LMs, contextual and isolated encoding, special-token inclusion, and single-layer versus layer-wise averaging.
- Table 1 enumerates 24 embedding-extraction configurations formed from combinations of pipeline components.
3 Experimental Setup
The experiments compare monolingual and multilingual BERT representations with fastText across six languages and five lexical evaluation tasks.
- The language sample comprises English, German, Russian, Finnish, Chinese, and Turkish, using monolingual uncased BERT Base models and multilingual BERT.
- Representations cover the top 100K frequent words in fastText vocabularies, enabling direct comparison with fastText static vectors.
- The study evaluates lexical semantic similarity, word analogy resolution, bilingual lexicon induction, cross-lingual information retrieval, and lexical relation prediction.
- LSIM uses Spearman correlation on Multi-SimLex, while WA uses Precision@1 on BATS with 99,200 analogy questions.
- BLI learns mappings from 5K training pairs and evaluates 2K test pairs with MRR across 10 language pairs.
- CLIR uses mapped cross-lingual vectors for document retrieval on CLEF 2003 and reports MAP across six language pairs.
4 Results and Discussion
Lexical extraction performance varies by language and task, but several patterns recur: monolingual models, contextual averaging, excluding special tokens, and lower-layer subsets generally help. Type-level lexical information is distributed across multiple layers, with lower layers carrying more of it, while cross-lingual performance remains mixed against fastText.
- Monolingual LMs contain more target-language lexical information than multilingual LMs, with larger gaps for Finnish and Turkish.
- External contexts generally improve lexical representations, although 100 contexts provide only marginal gains over 10 and some tasks show small differences.For MONO.*.NOSPEC.AVG(L≤8), EN–FI BLI rises from 0.486 with ISO to 0.503 with AOC, while DE–TR BLI rises from 0.315 to 0.334.
- Excluding [CLS] and [SEP] consistently improves type-level lexical embeddings across languages, source LMs, and layer ranges.NOSPEC configurations outperform both ALL and WITHCLS in ISO and AOC-{10, 100} settings.
- Averaging lower-layer subsets generally outperforms single-layer or all-layer extraction, indicating lexical information is distributed across layers but concentrated near the bottom.With MONO.ISO.NOSPEC, L≤6 averages score 0.561 in LSIM, 0.076 in CLIR, and 0.432 in BLI, versus 0.218, 0.008, and 0.230 for the top six layers.
- The best BERT-based configurations outperform fastText in several monolingual tasks, but fastText scores higher overall in BLI and CLIR.BERT-based vectors outperform fastText on LSIM for EN, FI, and ZH, on WA, and on RELP for EN and DE; cross-lingual comparisons are not strictly apples-to-apples.
5 Further Discussion and Conclusion
The discussion finds that model size and pretraining data can affect lexical knowledge, while cross-lingual performance varies by representation source and task. It concludes with configuration recommendations and several directions for extending the study.
- EN BERT-Large raises peak EN LSIM from 0.518 to 0.531 over BERT-Base.
- 81GB of Italian pretraining text improves EN–IT BLI from 0.548 to 0.572 and CLIR from 0.148 to 0.160.
- The study recommends monolingual LMs, contextual word encoding, and carefully selected extraction configurations for lexical representations.
- Future work will examine external-corpus domains, representative context sampling, more languages and probes, larger LMs, layer weighting, and multilingual enrichment.
A Appendix
The appendix documents model, corpus, evaluation, and task resources, and supplies expanded results for bilingual lexicon induction, retrieval, relation prediction, and bilingual layer correspondence.
- Figures 8–10 show bilingual layer correspondence through linear CKA heatmaps for additional English–Lithuanian language pairs.
- Table 3 lists URLs for models used in the main experiments and side experiments.
- Table 4 links the external corpora, from which 1M sentences of maximum length 512 are randomly sampled.
- Table 5 provides links to evaluation data and task models.
- Tables 6 and 7 report expanded BLI results across language pairs and extraction configurations, using MRR and AVG(L≤n) layer averaging.
- Table 8 reports CLIR results as MAP scores across language pairs and extraction configurations.
- Table 9 reports RELP micro-averaged F1 scores over five runs, including standard deviations.