Source-linked AI summary
Word Sense Disambiguation using a Bidirectional LSTM
Mikael Kågebäck, Hans Salomonsson
TL;DR
Word sense disambiguation must account for contextual ambiguity and word order, while existing systems may disregard order or rely on handcrafted features and external resources. This paper uses a shared bidirectional LSTM for WSD, achieving the top score on SE2 and tying IMS+adapted CW on SE3. Its conclusion describes the model as applicable to resource-poor languages, while noting that evaluation across languages and on large-vocabulary all-words WSD remains future work.
Problem
WSD assigns ambiguous word tokens to senses using context, but some systems disregard word order or rely on complex, potentially language-specific features and resources.
Source: §1 · p. 1Method
The model shares a bidirectional LSTM and hidden-layer parameters across word types, while using word-type-specific softmax parameters to predict senses.
Source: §3 · p. 2Results
The model achieves the top score on SE2 and ties IMS+adapted CW on SE3.
Source: §4.2 · p. 5Takeaways & Limitations
The authors describe the model as largely language independent and applicable to resource-poor languages, and designed to generalize to full-vocabulary WSD through parameter sharing.
Source: §5 · p. 5Evidence for language independence and evaluation on large-vocabulary all-words WSD remain planned future work.
Source: §5 · p. 5
Abstract
from arXiv · showhide
In this paper we present a clean, yet effective, model for word sense disambiguation. Our approach leverage a bidirectional long short-term memory network which is shared between all words. This enables the model to share statistical strength and to scale well with vocabulary size. The model is trained end-to-end, directly from the raw text to sense labels, and makes effective use of word order. We evaluate our approach on two standard datasets, using identical hyperparameter settings, which are in turn tuned on a third set of held out data. We employ no external resources (e.g. knowledge graphs, part-of-speech tagging, etc), language specific features, or hand crafted rules, but still achieve statistically equivalent results to the best state-of-the-art systems, that employ no such limitations.
1 Introduction
Word sense disambiguation depends on context, where word order can change a token’s sense; the introduction argues that WSD requires sequence modeling rather than bag-of-words context. The paper presents a purely learned system and contributions centered on parameter sharing, word-order evidence, and combining local and global information.
1 Introduction
- Current WSD systems may disregard word order or rely on complex, potentially language-specific features and resources, motivating sequence modeling with word embeddings instead.The introduction also notes that improved WSD would benefit several NLP tasks.Source: §1 · p. 1
1.1 The main contributions of this work include:
- The paper presents a purely learned WSD approach that achieves results on par with resource-heavy systems.Source: §1.1 · p. 2
- Parameter sharing across word types aims to use labeled data more efficiently and make full-vocabulary scaling plausible without exploding the parameter count.Source: §1.1 · p. 2
- The paper provides empirical evidence that word order matters for WSD.Source: §1.1 · p. 2
- With no explicit window, the system can combine local and global information when deducing a word’s sense.Source: §1.1 · p. 2
2 Background
The background introduces bidirectional LSTMs and GloVe word embeddings as techniques underlying the proposed model.
2.1 Bidirectional LSTM
- An LSTM is a gated recurrent neural network designed to capture long-term dependencies; a bidirectional LSTM combines left- and right-going states to represent preceding and succeeding words.The account attributes the LSTM to Hochreiter and Schmidhuber (1997) and the bidirectional variant to Graves and Schmidhuber (2005).Source: §2.1 · p. 2
2.2 Word embeddings by GloVe
- Word embeddings represent words as real-valued vectors in a semantically meaningful space and can initialize a neural network’s input layer.Source: §2.2 · p. 2
- GloVe combines a log-linear model with co-occurrence statistics to capture global statistics more efficiently, and its vectors can encode fine-grained semantic and syntactic information.Source: §2.2 · p. 2
3 The Model
The model uses a BLSTM and hidden layer shared across word types, with a word-specific softmax producing sense probabilities. Dropword regularizes training by replacing selected words with a trained tag that changes over time.
3.1 Model definition
- A shared BLSTM and hidden layer encode context, while a word-specific softmax produces a distribution over that word’s senses.The hidden representation concatenates the outputs of left- and right-traversing LSTMs at the target word.Source: §3 · p. 2, §3.1 · p. 3, Figure 1 · p. 3
- The BLSTM input is the target word’s embedding, selected from the embedding matrix using its one-hot word-type representation.The embedding matrix can be initialized with pre-trained vectors; this work uses GloVe.Source: §3.1 · pp. 2–3
3.2 Dropword
- Dropword replaces selected context words with a trained <dropped> tag, with the dropped words changing over time.The technique is intended to reduce dependence on individual words in the training context.Source: §3.2 · p. 3
4 Experiments
The experiments evaluate the model on two English lexical-sample tasks, with shared hyperparameter settings tuned on held-out data. The model ranks first on SE2 and ties IMS+adapted CW on SE3; results also support the importance of word order and pretrained embeddings.
4 Experiments
- The evaluation uses the SensEval 2 and 3 English lexical-sample tasks, each disambiguating one indicated word in context.The tasks provide training and test data, with gold-standard sense labels and target words sampled across low, medium, and high frequencies.Source: §4 · p. 4
4.1 Experimental settings
- Hyperparameters for both experiments were tuned on a separate validation set drawn from SE2 training data.Inputs were initialized with GloVe vectors, numbers were replaced with a tag, and context was capped at 140 words centered on the target; test words absent from training were treated as unknown.Source: §4.1 · p. 4, Table 1 · p. 4
4.2 Results
- The compared systems use rich feature sets, including raw words, lemmas, POS tags, and collocations, while IMS+adapted CW also uses separately trained embeddings.The proposed model's reported top and tied scores are against this state-of-the-art comparison set.Source: §4.2 · pp. 4–5
- The model achieves the top score on SE2 and ties IMS+adapted CW on SE3.Dropword consistently improves results on both datasets; randomizing input order substantially worsens results, and pretrained embeddings are essential to performance on these datasets.Source: §4.2 · p. 5, Table 2 · p. 5
5 Conclusions & future work
The paper concludes that a BLSTM model can use word order to achieve state-of-the-art-level WSD results without external resources or handcrafted features. It identifies multilingual and large-vocabulary all-words evaluation as future work.
5 Conclusions & future work
- The BLSTM model exploits word order and reaches state-of-the-art-level results without external resources or handcrafted features.The authors describe it as largely language independent and applicable to resource-poor languages.Source: §5 · p. 5
- The model shares most parameters between words to support generalization to full-vocabulary WSD.Source: §5 · p. 5
- Future work includes testing language independence across several languages and evaluating large-vocabulary all-words WSD.The authors also plan to test unsupervised BLSTM pretraining, encouraged by the improvement from word embeddings.Source: §5 · p. 5