PaperlayerView exact revision on arXiv

Source-linked AI summary

Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders

Terra Blevins, Luke Zettlemoyer

arXiv:2005.02590v2cs.CL

TL;DR

WSD struggles with rare and unseen senses because word-sense frequencies are imbalanced. The paper proposes a jointly optimized bi-encoder that embeds target context and sense glosses in one space, and reports improved all-words WSD performance, including a 31.1% error reduction on less frequent senses.

  • Problem

    Sense-frequency imbalance makes existing WSD models perform poorly on rare or unseen senses, creating a challenge for accurate contextual disambiguation.

  • Method

    The BEM independently embeds each target word with its context and each candidate sense gloss, jointly optimizing both encoders in the same representation space.

  • Results

    31.1% error reduction over prior work is reported on less frequent-sense examples, alongside improved English all-words and zero-shot performance.

  • Takeaways & Limitations

    Modeling sense definitions helps the bi-encoder disambiguate less frequent and zero-shot senses without sacrificing accuracy on the most common senses.

  • Takeaways & Limitations

    A large MFS–LFS performance gap remains, with the model performing over 40 points better on MFS examples.

Abstract

from arXiv · show

A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed, causing existing models to generally perform poorly on senses that are either rare or unseen during training. We propose a bi-encoder model that independently embeds (1) the target word with its surrounding context and (2) the dictionary definition, or gloss, of each sense. The encoders are jointly optimized in the same representation space, so that sense disambiguation can be performed by finding the nearest sense embedding for each target word embedding. Our system outperforms previous state-of-the-art models on English all-words WSD; these gains predominantly come from improved performance on rare senses, leading to a 31.1% error reduction on less frequent senses over prior work. This demonstrates that rare senses can be more effectively disambiguated by modeling their definitions.

1 Introduction

WSD systems face severe sense-frequency imbalance, motivating a jointly optimized BERT bi-encoder that uses context and sense glosses to improve rare- and unseen-sense disambiguation. The reported gains come primarily from less frequent senses while preserving performance on common senses.

  • Motivation: 90% of SemCor mentions of plant correspond to its top two senses, while only five of ten senses occur at all.This imbalance biases systems toward predicting the most frequent sense regardless of context.
  • Approach: The BEM jointly fine-tunes a context encoder and a gloss encoder end-to-end using only the WSD objective.It represents target words with surrounding context and sense definitions in the same embedding space.
  • Results: The model outperforms prior work on English all-words WSD, with gains coming almost entirely from better less frequent-sense performance.It also improves in the zero-shot setting for words and senses unseen during training.
  • Results: 15.6 absolute F1 improvement over the closest performing system is reported for less frequent senses.
  • Results: The bi-encoder generalizes better than a strong pretrained baseline from limited examples per sense, indicating data efficiency for less frequent senses.Few-shot training is used to study learning with limited training examples.
  • Contributions: The contributions include improved all-words English WSD and better LFS and zero-shot performance without sacrificing accuracy on the most common senses.

2 Background and Related Work

WSD predicts a word’s sense in context, with applications including machine translation and information extraction. Related systems use engineered features, neural encoders, glosses, pretrained models, or cross-encoded context–gloss classification, while this work uses independent bi-encoding.

  • Task: WSD predicts the particular sense of a word when it occurs in a specific context.The all-words formulation disambiguates every ambiguous word in the data.
  • Task: WSD has been shown to help downstream tasks such as machine translation and information extraction.
  • Prior approaches: Earlier systems used manually engineered features and independent classifiers or word experts for each ambiguous lemma.Later neural approaches used learned encoders, shared output spaces, or semi-supervised learning.
  • Gloss information: Gloss-based methods improve WSD by using sense definitions, extending from definition–context overlap to WordNet structure, embeddings, and neural inputs.
  • Gloss information: Kumar et al. represented senses from encoded glosses but froze them after knowledge-graph supervision, requiring an additional supervision source.The present approach instead jointly learns the encoders from the WSD objective alone.
  • Pretrained models: Pretrained models capture word-sense information, and prior work used BERT representations or contextualized outputs to create sense embeddings.
  • Proposed architecture: The proposed BEM independently encodes target context and sense-definition text with BERT-initialized encoders in a shared representation space.
  • Pretrained models: Cross-encoded context–gloss classification concatenates both texts and predicts whether a gloss matches the target sense.The paper contrasts this with independently encoding contexts and glosses, which is described as more computationally efficient.

3 Methodology

The BEM models WSD with two independently initialized BERT encoders: one represents a target word in context, while the other represents each sense from its gloss. Candidate senses are scored by similarity between the context and gloss representations, and training uses cross-entropy over candidates.

  • Architecture: BEM uses independent context and gloss encoders to represent target words and their candidate senses.Both encoders are initialized with BERT and jointly trained to place each target representation near its correct sense representation.
  • Context encoding: The context encoder produces a representation for each target word within the surrounding sentence.When a word is split into multiple BERT subwords, its representation is the average of those subword representations.
  • Gloss encoding: The gloss encoder represents each sense from the definition text, using the first encoder output as the sense representation.The first output corresponds to the input [CLS] token and serves as a global gloss representation.
  • Scoring: The model scores every candidate sense by taking the dot product between the target-word representation and each sense representation.At evaluation time, it predicts the candidate with the highest dot-product score.
  • Training: BEM is trained with cross-entropy over the scores of the candidate senses for each target word.The loss is defined for a word–sense pair and contrasts the correct candidate with the alternatives.

4 Experimental Setup

The experiments train BEM on SemCor, develop on SE07, and evaluate on four standard all-words WSD datasets using WordNet glosses. The study compares BEM with frequency-based, frozen-BERT, and recent state-of-the-art systems under a common evaluation framework.

  • Data: All sense glosses are retrieved from WordNet 3.0 for the experiments.The evaluation follows the established English all-words WSD framework.
  • Baselines: The comparison includes WordNet S1 and most frequent sense baselines, a frozen BERT-base classifier, and six recent state-of-the-art systems.The baselines cover knowledge-based predictions, pretrained representations, and gloss-informed models.
  • Evaluation: The reported evaluation metric is F1-score on English all-words WSD, including an ALL result formed by concatenating the datasets and development set.The study reports the best result from a comparable single-model setting using the same training data.
  • Model comparison: BEM independently encodes contexts and glosses, unlike GlossBERT’s single cross-encoder that jointly encodes them.The BEM encoders are initialized with BERT-base and optimized using a dot-product score with cross-entropy loss.

5 Evaluation

The evaluation compares BEM with baselines and prior systems on English all-words WSD, then examines performance by sense frequency and zero-shot conditions. BEM achieves the strongest overall results, with gains concentrated on less frequent and unseen senses.

  • Evaluation Design: The evaluation covers overall English all-words WSD, MFS, LFS, and zero-shot words and senses, using comparisons with baselines and prior work.Additional evaluation materials include pretrained-encoder comparisons and ablations of the bi-encoder architecture.
  • Rare Senses: 15.6 F1 improvement over the BERT baseline occurs on less frequent senses, while MFS accuracy remains approximately unchanged.BEM still has a 41.5-point MFS–LFS gap, but this is smaller than for BERT-base and EWISE.
  • Zero-shot Results: BEM and EWISE outperform the WordNet S1 baseline on zero-shot words, with BEM achieving a slightly higher F1 score.BERT-base matches WordNet S1 because it backs off to that baseline for unseen words.
  • Zero-shot Results: BEM outperforms both WordNet S1 and BERT baselines by at least 15 F1 points on zero-shot senses.The result supports using gloss text to learn sense representations that generalize to senses absent from training.

6 Analysis Experiments

The analysis examines which architectural choices and pretrained encoders contribute to BEM performance, alongside sense balancing and learned representations. Independent encoders perform best, while balancing improves less frequent senses at a cost to common-sense accuracy.

  • 6.1 Model Ablations: Independent context and gloss encoders perform best, while tying their parameters decreases SemEval2007 F1 by 0.4 points.Freezing the gloss encoder is more harmful than freezing the context encoder.
  • 6.1 Model Ablations: BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large achieve similar SemEval2007 development-set performance despite differing model sizes.RoBERTa-large slightly outperforms BERT-base, but BERT-base is selected for training efficiency.
  • 6.2 Balancing the Senses: Inverse-frequency loss weighting raises LFS performance but lowers MFS performance, producing an ALL-set F1 of 77.6 versus standard BEM.The balanced model also slightly improves zero-shot words and senses.
  • 6.2 Balancing the Senses: Weighting each candidate sense equally gives less common senses stronger training signal but sacrifices the sometimes helpful bias toward frequent senses.The authors identify stronger training signals for uncommon senses as a direction for future work.
  • 6.3 Learned Representations: t-SNE visualizations show more concentrated sense clusters for BEM than frozen BERT-base, with noun senses better separated than verb senses.The text attributes poorer verb separation most likely to their limited training data.

7 Few-shot Learning of WSD

The few-shot experiments test BEM with limited examples per sense against a frozen BERT classifier. BEM is more data-efficient, retaining performance better at very small k and matching the full baseline with at most five examples per sense.

  • Few-shot setup and results: BEM has a smaller performance drop at k=1 than the pretrained baseline and achieves similar performance to the full baseline with 5 or fewer examples per sense.Models are trained with k = 1, 3, 5, 10 examples per sense and compared with full-data training.
  • Few-shot setup and results: Both BEM and the frozen BERT classifier achieve higher F1 as the number of training examples per sense increases.The comparison uses the ALL evaluation set across different k values and full-data training.
  • Implications: Adding a few labeled examples of rare senses may be more effective than annotating more data without considering the sense distribution.The authors connect this consideration to extending WSD into new domains or languages.

8 Conclusion

The conclusion presents BEM as a jointly optimized bi-encoder for improving WSD on uncommon senses. It reports substantial gains on less frequent senses while noting that a large MFS–LFS gap remains and that further work is needed.

  • Contribution: BEM maps senses and ambiguous words into one embedding space and assigns each word the label of its nearest sense embedding.The context and gloss encoders are jointly optimized for this shared space.
  • Results: 31.1% error reduction over prior work is achieved on less frequent sense examples.The conclusion identifies improved uncommon-sense disambiguation as the central outcome.
  • Limitations: BEM still performs over 40 points better on MFS than LFS examples, leaving a substantial frequency-based performance gap.The conclusion notes that similar trends appear in recent WSD systems, including frozen BERT-base.
  • Future work: Future work includes stronger training signals for uncommon senses, alternative architectures for LFS, and improved few-shot WSD methods.These directions are linked to modeling rare senses and extending WSD into new domains.

A Additional Training Details

The additional training details specify the optimization setup for the frozen BERT baseline and BEM. BEM uses separate GPUs for its two encoders to support larger batch sizes.

  • BERT-base baseline: The frozen BERT-base classifier is trained for 100 epochs with learning rates from 5e−6 to 1e−4 and batch sizes of 32, 64, or 128.Both systems are implemented in PyTorch and optimized with Adam.
  • Bi-encoder Model (BEM): BEM is trained for 20 epochs with a 10,000-step warmup, context batch size 4, and gloss batch size 256.Its learning-rate search range is 1e−6 to 5e−5.
  • Bi-encoder Model (BEM): BEM optimizes its two encoders on separate GPUs to allow larger batch sizes.The model is initialized with bert-base-uncased.

B Additional Sense Embedding Explorations

Additional t-SNE visualizations compare frozen BERT representations with the BEM context encoder across selected senses of good, light, and run. They show improved clustering for some senses but persistent overlap or difficulty for closely related distinctions.

  • Visualization setup: The visualizations use t-SNE, compare frozen BERT outputs with BEM context-encoder outputs, and show up to six common senses for words occurring over 50 times in SemCor.The left side shows frozen BERT-base representations, while the right side shows BEM representations.
  • good: For good, BEM does not improve the overall representation, but better separates adj.2 from adj.1 at one edge of the cluster.Examples of adj.2 appear only in one edge of the adj.1 cluster.
  • light: For light, BEM produces more distinct representation clusters than BERT-base, although cluster edges still overlap.
  • run: For run, both frozen BERT and BEM struggle to distinguish the closely related verb.1 and verb.2 senses.The distinction is described as very fine-grained, with the relevant glosses shown in the legend.
Loading 2005.02590v2…