Source-linked AI summary
Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation
Daniel Loureiro, Alipio Jorge
TL;DR
WSD remains difficult, while traditional word embeddings conflate the distinct senses of a word and supervised approaches face limited sense annotations. The paper builds full-coverage WordNet sense embeddings from contextual representations, WordNet relations, and glosses, then evaluates them with nearest-neighbor WSD. It reports gains on cross-domain WSD and explores analyses using the resulting representations.
Problem
WSD remains challenging despite advances in representations, while static word embeddings conflate senses and supervised methods face limited sense-annotated data.
Source: §1 · p. 1Method
The method creates sense embeddings across WordNet by extending annotated signal through semantic relations and building sense-specific embeddings from contextualized glosses.
Source: §1 · p. 2, §4.2 · pp. 4–5, §4.3 · p. 5Results
On cross-domain WSD, the full-coverage method improves on the state of the art, adding 1.9% F1 over BERT k-NN with MFS.
Source: §1 · p. 2Takeaways & Limitations
The sense embeddings support k-NN WSD without MFS fallbacks and can be used for concept-level analyses of contextual embeddings and their NLMs.
Source: §1 · p. 2, §8 · p. 9When POS information is missing, many verb targets are assigned adjective or noun senses; the proposed solution does not address this problem.
Source: §5.2 · p. 7
Abstract
from arXiv · showhide
Contextual embeddings represent a new generation of semantic representations learned from Neural Language Modelling (NLM) that addresses the issue of meaning conflation hampering traditional word embeddings. In this work, we show that contextual embeddings can be used to achieve unprecedented gains in Word Sense Disambiguation (WSD) tasks. Our approach focuses on creating sense-level embeddings with full-coverage of WordNet, and without recourse to explicit knowledge of sense distributions or task-specific modelling. As a result, a simple Nearest Neighbors (k-NN) method using our representations is able to consistently surpass the performance of previous systems using powerful neural sequencing models. We also analyse the robustness of our approach when ignoring part-of-speech and lemma features, requiring disambiguation against the full sense inventory, and revealing shortcomings to be improved. Finally, we explore applications of our sense embeddings for concept-level analyses of contextual embeddings and their respective NLMs.
1 Introduction
WSD remains difficult because word-level representations conflate senses, while limited sense annotations have constrained supervised methods. The paper uses contextual embeddings to build full-coverage WordNet sense representations and reports improved cross-domain WSD results.
1 Introduction
- The introduction frames WSD as assigning a word its correct sense in context, a task that remains challenging despite advances in representations.Even the most-frequent-sense baseline has been difficult to surpass.Source: §1 · p. 1
- Traditional word embeddings combine a word’s different senses, whereas contextual embeddings produce representations suited to the context.Contextual embeddings use pretrained NLMs and can draw on arbitrarily long contexts.Source: §1 · pp. 1–2
- Earlier SemCor-derived sense embeddings covered only a small fraction of WordNet, leaving many instances to MFS fallbacks.Limited sense-annotated data is identified as a major obstacle for supervised WSD.Source: §1 · pp. 1–2
- On cross-domain WSD, full-coverage embeddings add 1.9% F1 over BERT k-NN with MFS and improve on the state of the art.The BERT k-NN approach with MFS itself surpasses previous systems; the full-coverage method further improves results without MFS fallbacks.Source: §1 · p. 2
- The method creates embeddings for all WordNet senses, enabling k-NN WSD without MFS fallbacks.The approach extends annotated signal through WordNet’s semantic relations.Source: §1 · p. 2
- Sense embeddings are also proposed for concept-level analyses of contextual embeddings and their NLMs.Source: §1 · p. 2
2 Language Modelling Representations
The section introduces distributional representations learned from NLMs, contrasting fixed word-type embeddings with contextual embeddings that vary according to context.
2 Language Modelling Representations
- Distributional semantic representations learned from unsupervised NLMs are widely used in NLP.Source: §2 · p. 2
2.1 Static Word Embeddings
- Static word embeddings learn vectors from fixed-length context windows, but conflate a word type’s distinct senses.FastText can compositionally represent word types unseen during training.Source: §2.1 · p. 2
2.2 Contextual Embeddings
- Contextual embeddings use a pretrained NLM to represent the same word type differently across contexts, typically using arbitrarily long token-prediction contexts.ELMo and BERT are examples; BERT achieved state-of-the-art results on 11 NLP tasks after task-specific fine-tuning.Source: §2.2 · p. 2
3 Word Sense Disambiguation (WSD)
The section reviews WordNet and prior WSD approaches, then presents a k-NN method that builds full-coverage sense embeddings from annotations and extends them with dictionary and morphological information.
3 Word Sense Disambiguation (WSD)
- The section frames WSD as assigning a word its contextually appropriate sense and relates the paper to work spanning supervised and knowledge-based approaches.It notes that supervised methods have traditionally performed best, while increasingly incorporating resources associated with knowledge-based approaches.Source: §3 · p. 3
3.1 Sense Inventory, Attributes and Relations
- WordNet is a semantic network of general-domain concepts linked by relations such as synonymy and hypernymy; its representation includes synsets, lemmas, senses, and part-of-speech information.Source: §3.1 · p. 3
3.2 WSD State-of-the-Art
- Prior WSD work spans word embeddings, LSTM-based methods, neural sequence models, and gloss or semantic-network approaches, while relying on MFS fallback and lemma/POS restrictions.The section describes Iacobacci et al. (2016), Yuan et al. (2016), Raganato et al. (2017b), Melamud et al. (2016), Peters et al. (2018), and Luo et al. (2018a, 2018b) and Vial et al. (2018).Source: §3.2 · p. 3
3.3 Other methods with Sense Embeddings
- Earlier sense-embedding methods use glosses, word embeddings, or semantic-network structure, but fixed spaces limit contextual matching and these methods lack full WordNet coverage.The proposed approach instead uses k-NN with full-coverage embeddings to avoid MFS fallbacks.Source: §3.3 · p. 4, §4 · p. 4, Figure 1 · p. 4
4.1 Embeddings from Annotations
- Sense embeddings are bootstrapped by averaging contextual embeddings of tokens labeled with each sense in annotated corpora.The paper uses pretrained ELMo and BERT models to generate the contextual embeddings.Source: §4.1 · p. 4
4.2 Extending Annotation Coverage
- Coverage extension across increasingly abstract WordNet representations improves recall and precision on the ALL test set.The procedure uses synset, hypernym, and lexname abstractions to impute missing sense embeddings.Source: §4.2 · p. 5, Table 1 · p. 5
4.3 Improving Senses using the Dictionary
- Dictionary-based sense embeddings average contextualized gloss representations and add the lemma specific to each sense alongside the synset lemmas.Concatenating dictionary and sense representations improves performance, which the authors attribute to preserving complementary gloss information.Source: §4.3 · p. 5, Table 2 · p. 6
4.4 Morphological Robustness
- FastText lemma embeddings make sense representations more resilient when lemma features are unavailable, and are crucial when candidate senses are not filtered.The method concatenates static lemma embeddings to its previous embeddings and aligns token embeddings in the same way.Source: §4.4 · p. 6, Table 2 · p. 6
5 Experiments
The experiments compare LMMS with current systems on cross-domain WSD and probe full-inventory matching, POS mismatches, and the use of world knowledge.
5.1 All-Words Disambiguation
- +1.9 F1 on ALL and +3.4 F1 on SemEval 2013: LMMS significantly improves on previous results, while BERT k-NN also surpasses prior systems.Table 3 compares systems on the Raganato et al. test sets.Source: §5.1 · p. 6, Table 3 · p. 7
- Figure 2 examines performance when additional neighbors count as valid predictions, alongside a random baseline for words with fewer senses than accepted neighbors.Source: §5.1 · p. 6, Figure 2 · p. 7
5.2 Part-of-Speech Mismatches
- In full-inventory matching, many verb targets are assigned adjective or noun senses; the approach addresses missing lemmas but not missing POS information.Source: §5.2 · p. 7, Table 4 · p. 7
5.3 Uninformed Sense Matching
- Uninformed Sense Matching (USM) searches the full set of more than 200K sense embeddings rather than a maximum of 59, making disambiguation harder than filtered WSD.The authors use this variant because lemma and POS filtering can obscure the impact of NLM representations on k-NN.Source: §5.3 · p. 7
5.4 Use of World Knowledge
- Controlled “played” examples match contextually appropriate senses, including riffing and performing at a location, suggesting entity topics help disambiguate verbs.The authors note these examples would be less conclusive without full WordNet coverage.Source: §5.4 · pp. 7–8, Table 5 · p. 8
6 Other Applications
Sense embeddings support concept-level gender-bias analyses of contextual language models, including differences between senses of the same lemma. They can also be used in applications that do not depend on a particular sense inventory.
6 Other Applications
- Sense embeddings make it possible to analyze gender bias at the synset level, including differences between senses sharing a lemma.Examples include doctor as PhD versus medic, and counselor as therapist versus summer camp supervisor.Source: §6 · p. 8
- Loureiro and Jorge (2019) use similarities between matched sense and contextual embeddings to train a classifier for whether a word has the same meaning across two sentences.Source: §6 · p. 8
- Positive values indicate bias toward man1, while negative values indicate bias toward woman1.Source: Figure 3 · p. 8, §6 · p. 8
7 Future Work
Future work will explore multilingual resources and semi-supervised refinement of sense embeddings, as well as downstream tasks that use relational knowledge linked through WordNet concepts.
7 Future Work
- The authors plan to use multilingual embeddings and glosses to improve sense embeddings and evaluate them on multilingual WSD.Source: §7 · p. 8
- They are considering iteratively fine-tuning their best embeddings by using them to automatically annotate corpora and repeating the process until convergence.Source: §7 · p. 8
- The authors expect the embeddings to be useful for downstream tasks such as Natural Language Inference that may benefit from relational knowledge linked through WordNet concepts.Source: §7 · p. 8
8 Conclusion
The paper reports improved cross-domain WSD performance using full-coverage WordNet sense embeddings. It also presents a 1-NN approach without MFS fallbacks or task-specific modelling, a full-inventory evaluation variant, and applications beyond WSD.
8 Conclusion
- The method yields a clear improvement over the current state of the art on cross-domain WSD tasks.Source: §8 · p. 9
- Sense embeddings combine contextual embeddings, semantic networks, and glosses to cover all WordNet senses, enabling 1-NN WSD without MFS fallbacks or task-specific modelling.Source: §8 · p. 9
- The paper introduces a WSD variant that matches contextual embeddings against all WordNet senses to examine the strengths and weaknesses of NLM representations.Source: §8 · p. 9
- The paper explores applications beyond WSD, including gender-bias analyses.Source: §8 · p. 9