Source-linked AI summary
Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings
Gregor Wiedemann, Steffen Remus, Avi Chawla, Chris Biemann
TL;DR
The paper asks whether contextualized word embeddings encode enough context-sensitive information to support interpretable word sense disambiguation. It applies nearest-neighbor classification to CWE vectors and finds that performance can exceed state of the art on two lexical-sample tasks, while BERT most clearly organizes polysemous words into distinct sense regions.
Problem
The paper investigates whether contextualized word embeddings can directly represent polysemy well enough to support word sense disambiguation.
Method
It uses cosine-distance k-nearest-neighbor classification over contextualized embeddings from Flair, ELMo, and BERT, with localized sense voting and an imbalance normalization heuristic.
Results
The approach reports improvements over state-of-the-art results for SensEval-2 and SensEval-3, while BERT forms the clearest separable sense clusters.
Takeaways & Limitations
BERT’s contextualized space most clearly captures distinct regions for different senses, supporting interpretable investigation of how embeddings encode polysemy.
Takeaways & Limitations
The nearest-neighbor approach is especially limited by sparse training data and cannot learn complex decision boundaries when embedding regions are not spherical.
Abstract
from arXiv · showhide
Contextualized word embeddings (CWE) such as provided by ELMo (Peters et al., 2018), Flair NLP (Akbik et al., 2018), or BERT (Devlin et al., 2019) are a major recent innovation in NLP. CWEs provide semantic vector representations of words depending on their respective context. Their advantage over static word embeddings has been shown for a number of tasks, such as text classification, sequence tagging, or machine translation. Since vectors of the same word type can vary depending on the respective context, they implicitly provide a model for word sense disambiguation (WSD). We introduce a simple but effective approach to WSD using a nearest neighbor classification on CWEs. We compare the performance of different CWE models for the task and can report improvements above the current state of the art for two standard WSD benchmark datasets. We further show that the pre-trained BERT model is able to place polysemic words into distinct 'sense' regions of the embedding space, while ELMo and Flair NLP do not seem to possess this ability.
1 Synonymy and Polysemy of Word Representations
Polysemy makes a word’s meaning context-dependent, motivating representations that mediate between word forms and meanings. Contextualized word embeddings address this by producing distinct vectors for tokens in context and can be used directly for WSD.
- Polysemy and WSD: Polysemy means that a word’s meaning changes with its context, while WSD identifies the intended sense from a fixed inventory.WordNet is described as the most commonly used English sense inventory, with more than 200K word-sense pairs.
- Representing meaning: Traditional string-based representations collapse different senses into the same word form, whereas distributional representations use context as an intermediate semantic representation.The distributional hypothesis links meaning to contextual occurrence and similarity of words appearing in similar contexts.
- Sense embeddings: Sense embeddings assign multiple vectors to one word, but downstream use still requires WSD to select the appropriate embedding from the sense inventory.The number of senses may be fixed by a parameter or derived automatically.
- Contextualized representations: Contextualized word embeddings produce distinct vectors for each token in context, supporting more realistic modeling of word meaning and context.Their advantages over static embeddings have been demonstrated in tasks including text classification and sequence tagging.
- Paper contribution: The paper uses contextualized embeddings directly for WSD with an interpretable k-nearest-neighbor classifier and compares three models across four benchmark datasets.The approach preserves training-example provenance and reports new state-of-the-art results for two datasets.
2 Related Work
Prior WSD systems use knowledge-based, supervised, or semi-supervised strategies, while contextualized embedding approaches differ in how they encode context. The paper tests whether these differences affect their ability to capture polysemy.
- WSD approaches: WSD systems are categorized as knowledge-based, supervised, or semi-supervised according to how they use resources, annotated data, and unlabeled corpora.All three types rely on a context representation to predict the correct sense.
- Previous WSD models: Earlier supervised and semi-supervised systems use context vectors, recurrent neural networks, label propagation, and compressed sense vocabularies to improve WSD.Some systems train on SemCor and the Princeton Annotated Gloss Corpus, while later work uses BERT wordpiece embeddings.
- Contextualized embeddings: Contextualized embeddings combine sentence-level and word-level semantics and have improved many downstream tasks relative to static embeddings.The paper investigates Flair, ELMo, and BERT as three prominent contextualization approaches.
- CWE models: Flair combines static word embeddings with left- and right-context vectors from character language models.Its contextualization has been applied successfully to sequence tagging tasks such as named entity recognition and part-of-speech tagging.
- CWE models: ELMo uses two bidirectional word-level recurrent models and combines their layer outputs through weighted element-wise summation.This differs from Flair’s character language models and concatenation scheme.
- CWE models: BERT provides contextualized token embeddings through an end-to-end self-attention transformer trained with masked language modeling.The paper therefore hypothesizes that contextualization methods differ in their ability to capture polysemy.
3 Nearest Neighbor Classification for WSD
The proposed WSD system uses a simple, interpretable nearest-neighbor classifier over contextualized embeddings. It localizes decisions to observed senses and modifies voting to address highly imbalanced training data.
- Classifier: The method applies non-parametric k-nearest-neighbor classification to contextualized embeddings so the training examples behind each decision remain directly inspectable.This provides interpretability compared with parametric classifiers such as support vector machines or neural models.
- Classifier: The classifier assigns labels by plurality vote among nearby labeled samples, using the simple non-parametric version to investigate embedding semantics.The one-nearest-neighbor case predicts the label of the closest training instance.
- Representation and distance: Cosine distance compares CWE vectors, and localized WSD considers only senses observed during training for the target word.Words are represented by their lemmas, mapping inflected forms such as “danced,” “dances,” and “dancing” to “dance.”
- BERT preprocessing: For BERT, the method averages wordpiece vectors belonging to the target word and concatenates the averaged vectors from its last four layers.This adapts the classifier to BERT’s subword tokenization.
- Imbalance handling: Sparse and imbalanced sense data can make larger k values favor majority classes, so voting is normalized with k′ = min(k,|V_s|).V_s is the set of CWEs for the least frequent training examples of a given word sense s.
4 Datasets
The evaluation covers four standard WSD benchmarks spanning lexical-sample and all-words settings. Their training-data availability and corpus properties differ substantially, affecting comparability and task difficulty.
- Evaluation sets: The experiments use four standard evaluation sets: lexical-sample SensEval-2 and SensEval-3, plus all-words SemEval 2007 Task 7 and Task 17.The lexical-sample datasets provide training and test sets, whereas the all-words datasets provide only test data.
- Evaluation sets: The two all-words tasks differ in sense granularity, and their documents substantially overlap.Their lack of accompanying training data requires using separate training resources.
- Evaluation protocol: Table 2 reports kNN WSD performance with k = 1 using F1%, with the best result for each test set marked in bold.The table supports comparison of the classifier across the four evaluation settings.
- Dataset properties: SensEval-2 and SensEval-3 provide many more examples per word and sense than SemCor or WNGT.This dataset-property comparison is summarized in Table 1.
5 Experimental Results
Experiments evaluate contextualized embeddings with nearest-neighbor WSD across benchmark tasks, examining performance, sparsity effects, embedding-space structure, and qualitative errors. BERT yields the strongest overall evidence of sense separation, while data sparsity and confounding contextual similarity limit kNN.
- Experimental setup: Two experiments compare pretrained embeddings with kNN and optimize k for BERT while qualitatively examining successes and failures.The first uses k = 1; the second tests multiple k values with BERT embeddings.
- Contextualized Embeddings: BERT and ELMo kNN outperform state of the art on SensEval-2, while BERT also outperforms other embeddings on SensEval-3.These comparisons use k = 1 nearest-neighbor classification.
- Nearest Neighbors: More CWEs per word and sense improve performance on SE-2 and SE-3, even when SE-3 has a higher average number of senses per word.The all-words tasks provide less matching training data and differ more strongly from the test data.
- Senses in CWE space: BERT forms clearer sense clusters than ELMo or Flair, supporting kNN decisions, while kNN remains limited by sparse data and complex non-spherical boundaries.Flair senses are scattered, ELMo separates major senses slightly, and BERT produces some clearly separable clusters.
- Nearest Neighbors: Nearest neighbors can exploit lexical or semantic overlap, but those same similarities can produce false predictions, including confusion between related bank senses.A grass-bank example is misclassified because military-context similarity outweighs explicit lexical overlap elsewhere.
6 Conclusion
The paper tests whether contextualized word embeddings can distinguish word senses using an interpretable nearest-neighbor approach. CWEs capture senses generally, with BERT producing the clearest clustering, while future work targets newer models and sparse-data classification.
- The study evaluates Flair NLP, ELMo, and BERT contextualized embeddings for WSD using k-nearest neighbor classification across four benchmark datasets.The approach places contextualized token representations into regions of a shared vector space and examines nearest examples.
- The approach reports improvements over state-of-the-art results on the SenseEval-2 and SenseEval-3 lexical sample tasks.
- CWEs generally place differently used words in different regions of the embedding space, indicating that they capture sense information.
- BERT shows the strongest separation, with example instances forming clusters in the embedding space.
- Future work will evaluate XLM, RoBERTa, and XLNet and test more powerful classifiers for near-miss errors in sparse training settings.