Source-linked AI summary
SenseBERT: Driving Some Sense into BERT
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, Yoav Shoham
TL;DR
Word-form self-supervision can leave contextual word meaning underrepresented. SenseBERT adds WordNet-based weak supervision to BERT by predicting masked-word supersenses alongside masked words, achieving improved lexical understanding across supersense disambiguation and WiC, including state-of-the-art WiC performance.
Problem
Existing self-supervision operates at the word-form level, although word forms can represent multiple contextual senses.
Method
SenseBERT jointly predicts masked words and WordNet supersenses during BERT pre-training, using weak supervision from an expert-constructed sense inventory without human annotations.
Results
SenseBERT improves supersense disambiguation over vanilla BERT and achieves state-of-the-art WiC performance, with SenseBERTLARGE scoring 72.14, 2.5 points above BERTLARGE.
Takeaways & Limitations
The results indicate that semantic signals beyond the lexical level can be introduced during pre-training for models to acquire additional insight without human supervision.
Takeaways & Limitations
The soft-labeling objective can make the model overconfident in a strict subset of allowed senses, especially early in training.
Abstract
from arXiv · showhide
The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the Word in Context task.
1 Introduction
Existing self-supervision predicts word forms, which can obscure the multiple senses expressed by an ambiguous form. SenseBERT adds weakly supervised supersense prediction to BERT, using WordNet without human sense annotations.
- 1 Introduction: BERT masks words and trains the model to predict them from their surrounding context.
- 1 Introduction: Word forms are only surrogates for contextual meaning, creating ambiguity when one form has multiple senses.
- 1 Introduction: SenseBERT jointly predicts masked word forms and their meanings through an auxiliary masked-word sense prediction task.
- 1 Introduction: The model uses WordNet supersenses as coarse-grained weak labels, avoiding the need for human sense-annotated data.
- 1 Introduction: Words with multiple possible supersenses receive soft labels allowing the model to predict any of their listed senses.
- 1 Introduction: A single SenseBERTLARGE model reaches 72.14 on WiC, improving BERTLARGE by 2.5 points.
2 Related Work
Prior work explored sense information in static and contextualized embeddings, but vanilla BERT often fails to capture lexical semantics. SenseBERT addresses this gap with an explicit weakly supervised word-sense signal.
- 2 Related Work: Static embeddings and separate sense embeddings have incorporated word-sense information using inventories, hypernym relations, or sense glosses.
- 2 Related Work: Contextualized embeddings made sense-aware representations promising because word embeddings depend on context rather than being precomputed.
- 2 Related Work: Despite this promise, vanilla BERT often exhibits high supersense misclassification rates under word-level self-supervision.
- 2 Related Work: SenseBERT’s weakly supervised word-sense signal significantly bridges the lexical-semantics gap identified for vanilla BERT.
- 2 Related Work: SenseBERT improves WiC lexical-semantics performance even relative to models that incorporate WordNet-based linguistic knowledge.
3 Incorporating Word-Supersense Information in Pre-training
SenseBERT augments BERT with a parallel supersense-level prediction objective and injects supersense information into Transformer inputs. WordNet-derived allowed supersenses provide weak labels, while regularization and vocabulary expansion address noisy labels and rare-word coverage.
- 3.2 Weakly-Supervised Supersense Prediction Task: SenseBERT adds a parallel external mapping S to predict masked-word supersenses alongside BERT’s word-form language model.The supersense classifier maps contextualized Transformer outputs into a 45-category supersense vocabulary.
- 3.2 Weakly-Supervised Supersense Prediction Task: For each vocabulary word, A(w) is the union of WordNet supersenses associated with its lemmatized synsets and serves as the allowed-label set.Short words and stop words are excluded from supersense labeling through an empty allowed set.
- 3.2 Weakly-Supervised Supersense Prediction Task: The allowed-senses loss treats every allowed supersense as a valid target, using soft labeling to accommodate words with multiple possible supersenses.Repeated contextual examples are expected to reinforce the correct supersense while weakening signals from incorrect allowed labels.
- 3.2 Weakly-Supervised Supersense Prediction Task: A uniform-distribution regularizer counteracts collapse toward a strict subset of allowed supersenses during training.The regularization term is combined with the allowed-senses objective to form the semantic language-model loss, alongside BERT’s word-level loss.
- 3.3 Supersense Aware Input Embeddings: Sense-aware input embeddings add supersense information through the fixed word-to-supersense mapping M, while retaining BERT’s positional embeddings and tied word mapping W.The architecture omits only the jointly trained Next Sentence Prediction task from the illustration.
- 3.4 Rare Words Supersense Prediction: The 60K-vocabulary alternative adds 30K frequency-selected words to BERT’s original 30K vocabulary, but sub-word masking receives only the regular word-level loss.This alternative increases whole-word coverage for which supersense prediction is meaningful.
4 Semantic Language Model Visualization
SenseBERT exposes masked-word meanings through a supersense prediction head and represents supersenses in semantically organized spaces. Its predictions assign meaningful categories to masked positions and support lexical-semantic abstractions beyond surface wording.
- SenseBERT’s supersense mapping acts as an additional head that predicts a word’s supersense from context.
- The model jointly predicts a missing word’s meaning alongside the standard word-form language model.
- UMAP visualizations of supersense vectors show clustering by supersense part of speech and finer-grained semantic groupings.
- SenseBERT produces distributions over supersenses for masked positions, assigning semantically meaningful categories across contexts.
- Figure 4 examples show SenseBERT predicting the correct SemEval-SS and WiC labels where BERT failed.
- The sentence “Dan cooked a bass on the grill” is abstracted as “[Person] [created] [food] on the [artifact]”, illustrating lexical-semantic generalization.
5 Lexical Semantics Experiments
SenseBERT is evaluated on lexical-semantic tasks, including supersense disambiguation, WiC, and GLUE, with comparisons across prediction methods and training schemes.
- The evaluations target supersense disambiguation, Word in Context, and GLUE, testing lexical-semantic categorization and broader language understanding.SemEval-SS and WiC directly assess lexical semantics, while GLUE tests whether semantic gains compromise other downstream performance.
- 5.1 Comparing Rare Words Supersense Prediction Methods: Both rare-word supersense methods perform comparably and improve over learning supersense information only for BERT’s original 30K-token vocabulary.The experiments continue with the 60K-token vocabulary while identifying average subword embeddings as a viable alternative.
- 5.2 SemEval-SS: Supersense Disambiguation: SenseBERTBASE improves over BERTBASE by over 10 points and SenseBERTLARGE improves over BERTLARGE by over 12 points in the Frozen SemEval-SS setting.Frozen training uses a linear classifier over pretrained embeddings without changing the model parameters.
- 5.2 SemEval-SS: Supersense Disambiguation: In fine-tuning, SenseBERTBASE surpasses BERTLARGE by 2 points on SemEval-SS, while SenseBERTLARGE surpasses vanilla BERTLARGE on WiC.Fine-tuning allows network parameters to change during task training, measuring the model’s ability to acquire word-supersense information after pre-training.
- 5.4 GLUE: SenseBERT performs on par with BERT on GLUE, achieving 77.9 overall versus 77.5 for BERTBASE.Both models were trained with the same data and fine-tuned using identical procedures and hyperparameters.
6 Conclusion
The paper introduces lexical-semantic information into neural language-model pre-training through SenseBERT. The resulting model improves word-level semantic awareness without human annotation and achieves strong lexical-semantic task performance.
- SenseBERT adds lexical semantic information to a neural language model’s pre-training objective.The approach uses an external linguistic knowledge source rather than human annotation.
- The model considerably outperforms vanilla BERT on SemEval-based Supersense Disambiguation and achieves state-of-the-art results on Word in Context.
- The results indicate that semantic signals beyond the lexical level can be introduced during pre-training without human supervision.
A Supersenses and Their Representation in SenseBERT
SenseBERT learns representations for WordNet supersenses, whose structure can be examined through clustering of the learned embedding vectors.
- Agglomerative clustering of SenseBERT’s learned supersense vectors clearly separates noun and verb senses.
- Semantically related supersenses, such as noun.animal and noun.plant, cluster together.
B Training Details
The appendix reports fine-tuning hyperparameters and provides visual and categorical references for SenseBERT’s supersense representations.
- B Training Details: Fine-tuning used maximum sequence length 128, learning rates from 5e−6 to 5e−5, batch sizes 16 or 32, and up to 10 epochs.
- A Supersenses and Their Representation in SenseBERT: Figure 5 visualizes an agglomerative hierarchical clustering over SenseBERT’s learned supersense vectors.
- A Supersenses and Their Representation in SenseBERT: Table 5 lists the WordNet supersense categories used in the paper.