Source-linked AI summary
SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection
Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, Nina Tahmasebi
TL;DR
Lexical Semantic Change detection lacks standard evaluation tasks and gold-standard data, hindering comparable progress. The paper presents the first SemEval shared task with multilingual manually annotated datasets and two subtasks, finding that type embeddings surprisingly outperform token embeddings while both subtasks remain far from solved.
Problem
Lexical Semantic Change detection lacks standard evaluation tasks and manually annotated diachronic data for producing high-quality, comparable results.
Method
The paper evaluates unsupervised Lexical Semantic Change systems on two subtasks across English, German, Latin, and Swedish using approximately 100,000 human judgments.
Results
Type embeddings outperformed token embeddings on both subtasks, while systems performed differently across languages and both subtasks remained far from solved.
Takeaways & Limitations
The shared benchmark enables more thorough testing of prior conclusions and comparison of future Lexical Semantic Change models.
Takeaways & Limitations
Agreement was negatively affected because later annotation rounds contained uses made harder to judge by propagated disagreements.
Abstract
from arXiv · showhide
Lexical Semantic Change detection, i.e., the task of identifying words that change meaning over time, is a very active research area, with applications in NLP, lexicography, and linguistics. Evaluation is currently the most pressing problem in Lexical Semantic Change detection, as no gold standards are available to the community, which hinders progress. We present the results of the first shared task that addresses this gap by providing researchers with an evaluation framework and manually annotated, high-quality datasets for English, German, Latin, and Swedish. 33 teams submitted 186 systems, which were evaluated on two subtasks.
1 Overview
Computational Lexical Semantic Change detection lacks standardized evaluation tasks and manually annotated diachronic data. SemEval-2020 Task 1 addresses this gap with multilingual subtasks, gold standards, and analysis of submitted systems.
- 1 Overview: The field lacks standard evaluation tasks and data, limiting high-quality, comparable results for Lexical Semantic Change detection.Most studies differ in evaluation procedures and factors considered, while few use manually annotated diachronic corpora.
- 1 Overview: The shared task introduced two related unsupervised subtasks for identifying words whose meanings change over time using corpus data.
- 1 Overview: The multilingual gold standard covers English, German, Latin, and Swedish and relies on approximately 100,000 human judgments.
- 1 Overview: The task enables comparisons across languages and investigates semantic representations, alignment methods, and change measures.The analysis also uncovers model trends and perspectives for further improvements.
- 1 Overview: The CodaLab website remains open so readers can compare their results directly with participating systems.The authors hope the task encourages study of Lexical Semantic Change in more languages, especially less-resourced ones.
2 Subtasks
The task simplifies Lexical Semantic Change detection by comparing two time-specific corpora while evaluating both categorical sense addition or loss and graded distributional change. These subtasks capture complementary notions of semantic change.
- 2 Subtasks: The task compares two time-specific corpora, reducing annotation demands and task complexity while allowing larger samples and more model architectures.
- 2 Subtasks: Subtask 1 asks systems to classify target words as having lost or gained senses between C1 and C2, or as unchanged.
- 2 Subtasks: Subtask 2 asks systems to rank target words by their degree of Lexical Semantic Change between C1 and C2.
- 2 Subtasks: Binary change detects whether a sense is lost or gained, whereas graded change captures frequency shifts even when a sense is not completely lost.For example, a newly acquired sense counts as binary change, while a substantial but incomplete frequency drop contributes to graded change.
- 2 Subtasks: The binary notion is especially relevant to historical linguistics and lexicography.
3 Data
The benchmark uses an unsupervised setup with diachronic corpus pairs, target words, and hidden manually assigned binary and graded labels across four languages. Test corpora are divided into two time periods and standardized before release.
- 3 Data: Participants received trial and test data but no training data, and predictions were evaluated against hidden gold labels.Models were trained only on the task corpora, although unsupervised pre-trained embeddings were allowed.
- 3 Data: The corpora cover English, German, Latin, and Swedish historical or contemporary time spans drawn from multiple named sources.
- 3 Data: Each corpus was divided into two time-specific subcorpora, C1 and C2, based on data size and target-word availability.
- 3 Data: The released test corpora contain standardized preprocessing, including sentence-length filtering, selected downsampling, lemmatization, punctuation removal, and within-period shuffling.
- 3 Data: Target words include changing and stable words selected using etymological and historical dictionaries and further manual checks.
- 3 Data: Subtask 1 uses manually assigned binary labels, while Subtask 2 uses graded labels from 0 to 1 derived from sense-frequency distributions.
4 Annotation
The annotation constructs representative usage graphs and clusters them into senses, yielding binary and graded lexical semantic change measures across diachronic corpora. The resulting multilingual benchmark combines substantial human annotation with agreement, clustering, and dataset-balance analyses.
- Annotation setup: Large, representative samples were prioritized for a limited number of target words rather than annotating many words.
- Annotation setup: Annotators sampled up to 100 uses from each corpus per target word and incrementally annotated graph edges across several rounds.The usage graph was updated after each round, and subsequent edges were sampled from the resulting clustering.
- Annotation setup: The annotation covered English, German, and Swedish with native-speaker annotators, while Latin used annotators with high-level Latin knowledge because native speakers were unavailable.Latin annotation involved 10 annotators ranging from undergraduate students to senior researchers.
- Clustering: Weighted usage graphs were clustered by minimizing disagreements, combining negative within-cluster edges with positive across-cluster edges.The method uses a variation of correlation clustering for weighted, undirected, sparsely observed, noisy graphs.
- Clustering: Simulated Annealing approximated the global optimum because the optimization is NP-hard but the graphs contain at most 200 nodes.
- Clustering: The clustering approach infers the number of clusters, handles missing edges, and reduces the impact of erroneous judgments through global graph information.
- Results: Polysemy correlated moderately to highly with graded change scores because the test sets were not controlled for polysemy, while model predictions were only moderately biased toward it.
5 Evaluation
The task evaluates unsupervised lexical semantic change systems across two subtasks and four languages using hidden labels, standardized metrics, and explicit baselines.
- Evaluation protocol: Each submission is scored by averaging performance across English, German, Latin, and Swedish, with the leaderboard hidden during evaluation.Teams could submit up to 10 systems, of which the best was retained for final ranking.
- Evaluation metrics: Subtask 1 uses accuracy for binary change classification, while Subtask 2 uses Spearman’s ρ for ranking graded semantic change.Accuracy was chosen anticipating balanced classes; F1 is additionally reported for imbalanced languages.
- Baselines: The Freq. Baseline measures absolute normalized frequency differences, while the Count Baseline compares aligned corpus count vectors using cosine distance.Subtask 1 also includes a majority-class baseline that always predicts no change.
6 Participating Systems
The shared task attracted many systems built from semantic representations, alignment techniques, and change measures, with token and type embeddings as the main representation choices.
- Participation: Thirty-three teams submitted 186 systems, and detailed descriptions of participating systems are provided in the appendix.The systems totaled 53 team members.
- System components: Participating models combine a semantic representation, an alignment technique, and a change measure.Representations mainly use average type embeddings or contextualized token embeddings; token models often add clustering.
7 Results
The shared-task results reveal strong cross-language and representation effects: type embeddings generally outperform token embeddings, while system success varies substantially by language and subtask. Predictions also show measurable frequency and polysemy biases, and word difficulty differs between classification and ranking.
- UG Student Intern performs best in Subtask 2, while UWB leads Subtask 1 averaged across languages.
- Top-performing systems use static type embeddings and distinguish themselves mainly through alignment methods and post-processing such as feature extraction, ensembles, and threshold selection.Reported alignment methods include Canonical Correlation Analysis, Orthogonal Procrustes, and Temporal Referencing.
- System performance differs substantially across languages, so systems excelling in one language do not necessarily perform well in another.The authors relate this discrepancy to corpus size, corpus characteristics, resource availability, and the challenges of Latin data.
- Type embeddings overwhelmingly outperform token embeddings: 7 of the best 10 Subtask 1 systems and 8 of the best 10 Subtask 2 systems use type embeddings.The authors suggest that token-embedding usage conventions and preprocessing may not yet be optimal.
- Type-based predictions are strongly frequency-influenced, with correlations exceeding 0.8 for some models, whereas token embeddings show weaker frequency effects.The passage attributes the weaker token-embedding effect partly to their not being trained on the test corpora.
- Model predictions are only moderately biased toward polysemy, while prediction difficulty reverses across subtasks: changed words are harder in Subtask 1, but stable words are harder in Subtask 2 in most languages.For English, Subtask 1 words with gold class 1 have almost twice the average error of gold-class-0 words.
8 Conclusion
The first SemEval shared task established a multilingual benchmark for unsupervised lexical semantic change detection using approximately 100,000 human judgments. Results show that both subtasks remain challenging, while type embeddings outperform token embeddings and frequency bias remains a key issue for future models.
- The task evaluated a wide range of systems on two subtasks and four languages using a dataset based on approximately 100,000 human judgments.
- Both subtasks remain far from solved, although several teams achieved high performance on both.
- Type embeddings surprisingly outperformed token embeddings on both subtasks.The authors suspect that token embeddings’ potential has not fully unfolded because canonical application concepts and preprocessing remain underdeveloped.
- Avoiding corpus-derived frequency bias is an important challenge for future type-based models.
- The benchmark provides a standard evaluation framework and high-quality datasets for comparing future models and testing earlier conclusions more thoroughly.The authors note that the test sets are limited in size but remain useful for shared evaluation.
A.1 Edge sampling
The annotation pipeline reduces the number of usage-graph edges through iterative sampling, clustering, combination, exploration, and disagreement-resolution heuristics.
- Edge sampling: Annotating every pair would require n∗(n−1)/2 edges, so the pipeline samples edges while preserving information for clustering.The graph is updated after each annotation round and reclustered.
- Edge sampling: The first round samples 10% of uses and 30% of their edges by random walk, producing a connected reference subgraph.This design avoids inferring more clusters than the data contains before later rounds.
- Edge sampling: Subsequent rounds combine unassigned single uses with multi-clusters and explore 30% of edges among non-assignable uses.These steps progressively populate the graph and discover additional clusters.
- Edge sampling: Additional heuristics sample corroborating edges, redistribute substantial annotator disagreements, and refine the inferred graph structure.The heuristics were added after the first round.
A.3 Simulation
The paper validates its annotation and clustering procedures through simulation, then situates them among diverse shared-task systems using static, contextual, aligned, and clustered representations.
- A.3 Simulation: The validation simulates 40 ground-truth usage graphs with Zipfian sense frequencies spanning approximately 50–1000 occurrences.Change is introduced for half the target words by setting selected sense frequencies to zero in one corpus.
- A.3 Simulation: The simulated annotation adds normally distributed judgment error, and the resulting clustering is compared with the simulated ground truth.The simulation also evaluates feasibility and tunes node and edge sample sizes.
- A.3 Simulation: With the selected parameters, the annotation algorithm converges after approximately 5 rounds and 8,000 judgments per annotator on average.These requirements remained within the study’s time and budget constraints.
- Systems: Shared-task systems represented word uses with contextual BERT embeddings or words with static embeddings, then measured change using distances, divergences, or clustering.Examples include cosine distance, KL-divergence, Jensen-Shannon divergence, k-means, GMM, and HDBSCAN.
- Systems: Several systems aligned static embeddings across corpora using Orthogonal Procrustes or vector initialization before applying cosine distance.Other approaches combined SGNS, PPMI, BERT, temporal referencing, and distribution-based thresholds.
- Systems: One ensemble systematically combined existing representations, alignment methods, and similarity or dispersion measures before selecting or weighting model decisions.The approach used unsupervised thresholds, certainty filtering, majority voting, or weighted score averaging.
C Results with F1, Precision and Recall
Subtask 1 results are reported with precision, recall, and F1 for baseline and submitted systems, including averages across languages.
- Results: Table 7 reports precision, recall, and F1 scores for Subtask 1 binary classification.The table includes baseline systems and systems with submitted description papers.
- Results: The results section directs readers to Table 7 for participants’ Subtask 1 performance.The cited passage does not provide individual score values.
- Results: Systems are ordered by decreasing F1 of their best submission averaged over all languages.‘Avg.’ denotes the average across languages for each system.