Source-linked AI summary
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Mohammad Taher Pilehvar, Jose Camacho-Collados
TL;DR
Word embeddings are typically static, while word meanings can vary with context, and generic benchmarks for evaluating this dynamic behavior are scarce. The paper introduces WiC, an expert-curated binary classification benchmark built from lexicographic examples. WiC is challenging: BERTlarge improves about 15.5% over a random baseline, leaving an approximately 15% gap to the 80.0% human-level upperbound.
Problem
Static embeddings cannot represent context-dependent meanings, while existing generic benchmarks for evaluating dynamic word semantics are scarce and SCWS has important limitations.
Method
The paper constructs WiC as an expert-curated binary classification dataset from lexicographic examples, testing whether two contextual occurrences of a target word have the same meaning.
Results
BERTlarge improves around 15.5% absolute over a random baseline, while the best systems remain approximately 15% below the 80.0% human-level upperbound.
Takeaways & Limitations
WiC provides a challenging benchmark for evaluating context-sensitive word representations and leaves substantial room for future modeling work.
Takeaways & Limitations
SCWS has low annotation agreement and often uses different target words across context pairs, allowing context-independent models to be evaluated.
Abstract
from arXiv · showhide
By design, word embeddings are unable to model the dynamic nature of words' semantics, i.e., the property of words to correspond to potentially different meanings. To address this limitation, dozens of specialized meaning representation techniques such as sense or contextualized embeddings have been proposed. However, despite the popularity of research on this topic, very few evaluation benchmarks exist that specifically focus on the dynamic semantics of words. In this paper we show that existing models have surpassed the performance ceiling of the standard evaluation dataset for the purpose, i.e., Stanford Contextual Word Similarity, and highlight its shortcomings. To address the lack of a suitable benchmark, we put forward a large-scale Word in Context dataset, called WiC, based on annotations curated by experts, for generic evaluation of context-sensitive representations. WiC is released in https://pilehvar.github.io/wic/.
1 Introduction
Static word embeddings cannot represent context-dependent meanings, motivating specialized representations and exposing a need for better benchmarks. WiC is proposed as a high-quality benchmark for evaluating context-sensitive word representations.
- Motivation: Static word embeddings assign each word one context-independent representation, so they cannot reflect ambiguous words’ potentially different meanings.Specialized approaches include multi-prototype embeddings and contextualized embeddings that adapt representations to usage contexts.
- Benchmark gap: Research on context-sensitive embeddings lacks suitable generic benchmarks, while isolated-word similarity and downstream evaluations have important limitations.SCWS is identified as the only existing benchmark specifically focused on dynamic word semantics, but the paper highlights shortcomings for recent models.
- Contribution: WiC provides a high-quality benchmark for evaluating contextualized word and sense representations and word sense disambiguation.The dataset is framed as binary classification using identical words in different contexts, making context-insensitive models comparable to a random baseline.
- Contribution: WiC uses expert-curated annotations to evaluate whether two contextual occurrences of a word correspond to the same meaning.Its examples are presented as positive or negative pairs, as illustrated in the sample dataset table.
2 WiC: the Word-in-Context dataset
WiC constructs a binary word-in-context classification dataset from lexicographic examples, with filtering and quality checks designed to improve semantic clarity. Human evaluation reaches 80.0% accuracy, while the dataset remains challenging because of fine-grained distinctions and limited lexical overlap across splits.
- Task formulation: Each WiC instance contains a target noun or verb in two contexts, labeled according to whether both occurrences express the same meaning.Positive and negative instances depend on whether the contexts are listed for the same sense in the source resource.
- Dataset construction: WiC extracts contextual examples from WordNet, VerbNet, and Wiktionary, using BabelNet mappings to connect the latter resources to WordNet.Lexicographic examples provide the basis for constructing the dataset.
- Dataset construction: 23,949 WordNet, 10,564 Wiktionary, and 636 VerbNet initial examples were extracted before split construction and filtering.The development and test sets were constrained to avoid excessive instances for one target word and repeated contextual sentences.
- Pruning and quality control: The dataset removes problematic instances through semi-automatic checking and prunes subtle WordNet sense distinctions using graph and supersense relations.The pruning strategy addresses cases where fine-grained senses are difficult for humans to distinguish.
- Quality check: 80.0% average human accuracy establishes the estimated human-level performance upperbound, with annotators lacking target-resource sense distinctions.Individual annotator scores were 79%, 79%, 80%, and 82%, and overlapping annotations reached 80% agreement.
- Quality check: 57% accuracy on pruned instances versus 80% on the final set indicates that pruning improved semantic clarity.The pruned evaluation used 100 instances and produced individual scores of 56% and 58%.
- Statistics: The test split contains 1,256 unique target words and only 36% target-word overlap with training, limiting reliance on pattern matching.The training split contains 5,428 instances, and contextual sentences do not overlap across splits.
3 Experiments
The experiments evaluate multi-prototype, contextualized, sentence-level, and classifier-based systems on WiC. WiC is difficult: BERTlarge improves over random by about 15.5 percentage points, while substantial gaps remain between models and human performance.
- Systems and baselines: The experiments compare recent multi-prototype and contextualized word embeddings, sentence-level baselines, and two binary classifiers on WiC.The classifiers are an MLP and a cosine-distance threshold strategy; the MLP is trained with five runs to account for optimizer stochasticity.
- Results: BERTlarge achieves around 15.5% absolute improvement over the random baseline, but the dataset remains difficult for all techniques.The estimated human-level accuracy is 80.0%, whereas random performance is 50.0%.
- Results: The cosine-distance threshold classifier is more efficient than the MLP, which may not suit the relatively small training data.Both classifiers are applied to comparison systems except the LSTM baseline.
- Results: Context2vec is more competitive than ELMo among LSTM-based contextualized models, yet neither significantly improves over the simple sentence BoW baseline.The sentence BoW baseline also outperforms the sentence LSTM baseline.
- Results: DeConf is the best-performing multi-prototype technique, while DeConf and SW2V benefit indirectly from lexical-resource information.DeConf uses WordNet information, and SW2V leverages the larger BabelNet resource.
4 Related work
SCWS was designed to evaluate context-sensitive embeddings by pairing words with meaning-triggering contexts, but its annotation quality and task composition limit its suitability. In particular, context-independent models can exceed the dataset’s human-level performance upperbound.
- SCWS: SCWS contains 2,003 word pairs and estimates semantic similarity for words presented with contexts that trigger specific meanings.Its context-sensitive design distinguishes it from standard word similarity datasets such as RG-65 and Sim-Lex.
- Limitations: SCWS has low pairwise inter-rater agreement of 0.35 Spearman ρ, while mean inter-rater agreement is 0.52 as a human-level upperbound.The paper treats inter-rater agreement as a measure of annotation quality and rating homogeneity.
- Limitations: Because most SCWS instances use different target words, context-independent models can be evaluated on the dataset and may surpass its human-level upperbound.Google News Word2vec reaches 0.65 ρ, exceeding the optimistic inter-rater agreement upperbound of 0.52.
5 Conclusions
The paper presents WiC as a benchmark based on reliable lexicographic examples for evaluating whether models distinguish word meanings in context. Results leave substantial room for improving context-sensitive word semantics.
- Conclusions: WiC is a benchmark for evaluating context-sensitive word representations based on lexicographic examples.The paper presents lexicographic examples as a reliable basis for validating models’ ability to perceive and distinguish different word meanings.
- Conclusions: A considerable gap between state-of-the-art models and the human-level upperbound suggests that modeling word semantics in context remains challenging.The conclusion frames this gap as motivation for future work.