Source-linked AI summary
GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge
Luyao Huang, Chi Sun, Xipeng Qiu, Xuanjing Huang
TL;DR
WSD requires selecting the correct sense of an ambiguous word, while prior supervised and gloss-enhanced neural systems leave room for improvement. GlossBERT constructs context-gloss pairs and fine-tunes BERT-based classifiers, and it outperforms state-of-the-art systems on several English all-words benchmarks. The paper therefore supports sentence-pair classification as a way to leverage gloss knowledge in supervised WSD.
Problem
Prior supervised WSD methods rarely use lexical resources such as WordNet, while gloss-enhanced neural methods have achieved limited improvement over traditional word-expert systems.
Method
GlossBERT constructs context-gloss pairs from candidate WordNet senses, converts WSD into sentence-pair classification, and fine-tunes pre-trained BERT.
Results
GlossBERT outperforms state-of-the-art systems on several English all-words WSD benchmark datasets.
Takeaways & Limitations
Fine-tuning BERT on a sentence-pair formulation provides a way to better leverage gloss knowledge in supervised neural WSD.
Abstract
from arXiv · showhide
Word Sense Disambiguation (WSD) aims to find the exact sense of an ambiguous word in a particular context. Traditional supervised methods rarely take into consideration the lexical resources like WordNet, which are widely utilized in knowledge-based methods. Recent studies have shown the effectiveness of incorporating gloss (sense definition) into neural networks for WSD. However, compared with traditional word expert supervised methods, they have not achieved much improvement. In this paper, we focus on how to better leverage gloss knowledge in a supervised neural WSD system. We construct context-gloss pairs and propose three BERT-based models for WSD. We fine-tune the pre-trained BERT model on SemCor3.0 training corpus and the experimental results on several English all-words WSD benchmark datasets show that our approach outperforms the state-of-the-art systems.
1 Introduction
WSD seeks the exact sense of an ambiguous word in context, but existing supervised and gloss-enhanced neural methods retain important limitations. GlossBERT addresses this by combining context-gloss pairs with fine-tuned BERT and outperforms state-of-the-art systems on English all-words benchmarks.
- WSD identifies the exact sense of an ambiguous word in a particular context.
- Knowledge-based methods use WordNet glosses and semantic-graph structure, whereas traditional supervised methods train word-expert classifiers from designed features.
- Existing neural methods improve flexibility but do not completely surpass the strongest word-expert supervised systems.
- Gloss-enhanced neural approaches help supervised WSD but may not fully exploit gloss knowledge.
- GlossBERT constructs context-gloss pairs, proposes three BERT-based WSD models, and fine-tunes pre-trained BERT on SemCor3.0.
- GlossBERT outperforms state-of-the-art systems on several English all-words WSD benchmark datasets.
2 Methodology
The method formulates WSD around candidate senses from WordNet and uses BERT to classify context-gloss pairs. Three GlossBERT variants differ in how they represent or highlight the target word.
- WSD formulation: For each target word, WSD selects the most suitable candidate sense from a predefined inventory, usually WordNet.
- BERT: BERT is a multilayer bidirectional Transformer encoder pretrained with masked-language-model and next-sentence-prediction tasks, then fine-tuned for WSD.
- BERT baseline: BERT(Token-CLS) classifies each target using its final token representation, with a dedicated classification layer for every target lemma.
- GlossBERT: GlossBERT forms context-gloss pairs from all possible WordNet senses, converting WSD into a sentence-pair classification problem.
- Context-Gloss Pairs with Weak Supervision: Weak supervision adds signals that identify or emphasize the target word in the gloss and context sentences.
- GlossBERT prediction: Each target produces N labeled context-gloss instances; at test time, the sense with the highest probability of label = yes is selected.
- GlossBERT variants: GlossBERT(Token-CLS), GlossBERT(Sent-CLS), and GlossBERT(Sent-CLS-WS) use target-token, whole-sequence, and weakly supervised representations, respectively.
3 Experiments
The experiments evaluate GlossBERT on standard English all-words WSD benchmarks using SemCor3.0 training data and compare it with knowledge-based, supervised, neural, and BERT systems. Results show that exploiting gloss knowledge through BERT-based models substantially improves performance, with weak target-word highlighting performing best in most circumstances.
- Experimental setup: The experiments fine-tune uncased BERTBASE on SemCor3.0, using SE07 for selecting optimal settings.The model uses 12 Transformer blocks, 768 hidden units, 12 attention heads, and 110M parameters.
- Datasets: The evaluation uses five English all-words WSD benchmarks, with SE07 selected as the development set.The test sets are SE2, SE3, SE07, SE13, and SE15.
- Compared systems: Table 3 reports F1-score (%) for six blocks of systems, including MFS, knowledge-based, traditional supervised, neural, feature-based BERT, and GlossBERT models.Bold values identify the best single model trained only on SemCor.
- Results: The three GlossBERT models outperform other models by a substantial margin, while BERT(Token-CLS) without gloss information performs slightly worse than prior supervised and neural systems.The authors attribute the improvements to better use of gloss information rather than directly using BERT alone.
- Results: GlossBERT(Sent-CLS-WS) performs best in most circumstances, whereas GlossBERT(Token-CLS) outperforms GlossBERT(Sent-CLS).The comparison supports the importance of highlighting the target word, while weak highlighting may combine advantages of the other variants.
- Method: The GlossBERT models construct context-gloss pairs and recast WSD as sentence-pair classification, enabling one classifier instead of a dedicated classifier for each target lemma.This formulation combines the context with glosses for all possible WordNet senses.
4 Conclusion
The paper presents a supervised neural WSD solution that better leverages gloss knowledge by pairing contexts with candidate-sense glosses and fine-tuning BERT. On several English all-words benchmarks, the approach outperforms state-of-the-art systems.
- Conclusion: GlossBERT converts WSD into sentence-pair classification by constructing context-gloss pairs for candidate senses.The model fine-tunes pre-trained BERT on the SemCor3.0 training corpus.
- Conclusion: Experiments on several English all-words WSD benchmark datasets show that GlossBERT outperforms state-of-the-art systems.