Source-linked AI summary
Analysing Lexical Semantic Change with Contextualised Word Representations
Mario Giulianelli, Marco Del Tredici, Raquel Fernández
TL;DR
Lexical semantic change is difficult to model because word meaning is context-sensitive and often develops through polysemous stages. The paper uses BERT contextualised usage representations, clusters them into usage types, and measures change over time; these representations and detected shifts align positively with human judgements and capture varied linguistic phenomena.
Problem
Existing approaches struggle with lexical meaning that is continuous, context-modulated, and typically changes through polysemous stages rather than directly between senses.
Method
The method uses BERT to represent each word occurrence, clusters occurrences into usage types, and measures semantic change over time with several metrics.
Results
Model representations and detected semantic shifts are positively correlated with human intuitions, and qualitative analysis captures synchronic and diachronic linguistic phenomena.
Takeaways & Limitations
Contextualised word representations show potential for modelling and analysing lexical semantic change without lexicographic supervision.
Takeaways & Limitations
The metrics identify broadening but cannot capture the nature of emerging interpretations, such as metaphorisation.
Abstract
from arXiv · showhide
This paper presents the first unsupervised approach to lexical semantic change that makes use of contextualised word representations. We propose a novel method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements. Our extensive qualitative analysis demonstrates that our method captures a variety of synchronic and diachronic linguistic phenomena. We expect our work to inspire further research in this direction.
1 Introduction
Lexical semantic change matters for machine reading comprehension, but prior approaches struggle to represent context-sensitive meaning and polysemous transitions. The paper introduces an unsupervised BERT-based usage approach, new change metrics, and human-evaluated analyses.
- Lexical semantic change is relevant to computational linguistics because understanding individual word meanings underpins general machine reading comprehension.
- Aggregating all senses into one word representation is problematic because semantic change typically proceeds through polysemous stages.
- The proposed method uses BERT to represent each word occurrence contextually, clusters usages into types, and measures change over time.
- The paper presents the first unsupervised lexical semantic change approach using contextualised word representations and proposes several metrics for measuring change.
- The evaluation dataset contains human similarity judgements for more than 3K word usage pairs across different time periods.
- Model representations and detected semantic shifts are positively correlated with human intuitions, while qualitative analysis captures synchronic and diachronic phenomena.
2 Related Work
Earlier lexical semantic change research mainly used form-based or sense-based representations, each with limitations in temporal alignment, contextual detail, or fixed sense inventories. This paper applies contextualised representations to detect and analyse change without lexicographic supervision.
- Lexical semantic change detection has mainly followed form-based and sense-based approaches that model meaning through changes in linguistic distributions.
- Form-based methods compare independently trained or dynamically aligned word representations across time intervals to score semantic change.
- Sense-based models represent meaning as time-dependent distributions over senses, but commonly use bag-of-words contexts and manually fixed numbers of senses.
- Unsupervised alternatives avoid fixed sense counts, yet some require syntactically parsed diachronic corpora and focus exclusively on nouns.
- Contextualised representations encode usage-dependent lexical meaning and have performed well on usage similarity and word sense disambiguation tasks.
- This work investigates contextualised representations for detecting and analysing lexical semantic change without lexicographic supervision.
3 Method: A Usage-based Approach to Lexical Semantic Change
The method represents individual word usages contextually with BERT, clusters them into usage types, and measures semantic change across time using three metrics. It is designed to capture shifts involving changing distributions of coexisting usages while also supporting a clustering-independent distance measure.
- Usage representations: BERT computes a contextualised representation for every occurrence of a target word, preserving its sententially modulated meaning.The method uses hidden-layer activations at the target position, summed dimensionwise, and stores each occurrence’s context and temporal label.
- Usage types: K-Means partitions each word’s usage representations into similar usage types, with the number of clusters selected by maximising the silhouette score.The search considers K ∈ [2, 10], runs 10 initialisations for each K, and chooses the lowest-distortion clustering.
- Usage distributions: Normalised counts of usage types produce probability distributions for each time interval, making temporal changes in usage prominence measurable.The distribution records the relative frequency of each automatically induced usage type within an interval.
- Change metrics: Entropy difference and Jensen-Shannon divergence compare usage-type distributions, targeting changes such as broadening, narrowing, or shifts in cluster proportions.High entropy difference is expected to signal broadening, whereas negative values indicate narrowing; high JSD indicates very dissimilar distributions.
- Change metrics: Average pairwise distance measures distances between usage representations from different periods without relying on usage-type clustering.This avoids errors introduced by the clustering process; the method experiments with cosine, Euclidean, and Canberra distance.
- Temporal aggregation: For multiple intervals, mean summarises change across the whole period, whereas max identifies the successive intervals with the strongest shift.The JSD definition over more than two distributions is insensitive to temporal order and correlates less well with human ratings than pairwise metrics.
4 Data
The study analyses lexical change in COHA using balanced twentieth-century English data and the GEMS target words. It derives approximately 1.3 million contextualised usage representations for evaluation and further analysis.
- Corpus: COHA covers 1810–2009 across genres, but the study uses 1910–2009 because each decade provides at least 21M words.Earlier decades are discarded because their per-decade data are less balanced.
- Target words: The target vocabulary comprises 99 GEMS words with human semantic-shift scores from the 1960s to the 1990s.The original 100-word list excludes extracellular because it appears in only three COHA decades; the remaining words occur in at least eight decades.
- Processing: For each target occurrence, BERT supplies a contextualised representation that is clustered into usage types using K-Means and silhouette coefficients.The resulting collection contains approximately 1.3M usage representations from the second century of COHA.
5 Correlation with Human Judgements
The evaluation tests whether BERT-based usage similarities and semantic-change scores align with human judgements. Both evaluations find positive correlations, including significant correlations for the proposed change metrics.
- Usage-type evaluation: The usage-type evaluation compares BERT similarity scores between usage vectors with human similarity judgements using word-specific matrices and the Mantel test.Human judgements average five annotators’ ratings of same-word usage pairs shown with surrounding sentence context.
- Usage-type evaluation: 10 out of 16 words show significant positive human–BERT correlations, with ρ coefficients ranging from 0.13 to 0.45.The result indicates that BERT representations and similarity scores correlate with human similarity judgements.
- Semantic-change evaluation: The semantic-change evaluation compares scores from the three proposed metrics with human shift scores in GEMS for the 1960s–1990s.A relative-frequency baseline and correlations reported for two alternative approaches are included for reference.
- Semantic-change evaluation: All three proposed metrics yield significant positive correlations with GEMS scores, whereas the frequency-difference baseline is not significant.Table 1 reports Spearman’s ρ coefficients, with prior approaches provided as reference comparisons.
- Aggregation and metrics: The max aggregation across four time intervals performs best for all metrics, with APD using Euclidean distance in the reported table.APD can be as informative as JSD and ED despite not requiring clustering into usage types.
6 Analysis
The analysis shows that contextualised usage types capture diverse synchronic distinctions and reveal diachronic narrowing, broadening, and metaphorisation, while exposing clustering and metric limitations.
- What do Usage Types Capture?: Usage types distinguish polysemous and homonymous senses, literal and figurative meanings, syntactic roles, phrasal collocations, and named entities.Examples include curious, coach, curtain, sphere, refuse, iron curtain, and named-entity usages.
- What do Usage Types Capture?: Clusters can encode syntactic argument structure, separating nominal, transitive, intransitive, and infinitival usages of refuse.
- What do Usage Types Capture?: Some clusters reflect unexpected contextual distinctions, such as affirmative versus negative uses of doubt rather than noun–verb separation.
- What Kinds of Change are Observed?: Narrowing appears when usage types disappear or decline, as with coach after technological evolution; negative mean ED reliably indicates this pattern.ED can miss obsolescence when new usage types emerge simultaneously, as for tenure.
- What Kinds of Change are Observed?: Broadening occurs through emerging usage types, including optical-disk meanings and the historically situated metaphorical iron-curtain usage.The iron-curtain usage emerges in the 1930s–1940s, peaks in the 1950s, and remains low from the 1970s.
- What Kinds of Change are Observed?: JSD and APD best capture broadening, but their scores diverge when novel usages are rare or usage-type assignments contain errors.APD is less affected by address’s assignment errors, whereas JSD can detect low-frequency development in curtain.
- What Kinds of Change are Observed?: The metrics identify broadening but cannot determine the nature of emerging interpretations, such as whether a change is metaphorical.The authors leave inter-cluster source–target comparisons for future work.
7 Conclusion
The paper introduces an unsupervised contextualised-representation approach to lexical semantic change and finds alignment with human interpretation across varied linguistic phenomena.
- The authors present the first lexicographic-supervision-free approach to lexical semantic change using neural contextualised word representations.
- Representations and detected semantic shifts align with human interpretation, supported by a new dataset of human similarity judgements.
- The approach captures synchronic and diachronic phenomena while avoiding a fixed number of word senses and exposing morphosyntactic usage properties through example sentences.
A Appendix
The appendix provides supplementary materials related to the paper’s similarity-judgement evaluation.
- The appendix contains supplementary materials related to Section 5.1.
A.1 New Dataset of Similarity Judgements
The evaluation dataset pairs sampled usages from 16 target words and gathers multiple crowdsourced similarity judgements for each pair.
- For each of 16 target words, five usages are sampled from each usage type across 20-year COHA periods, producing 3,285 usage pairs.
- Each usage pair receives five similarity judgements from Figure Eight annotators, with usages presented in their surrounding sentence context.Each usage includes the target sentence plus preceding and following sentences, averaging 67 tokens.
- The appendix documents annotator instructions and illustrates a single Figure Eight annotation item.
A.2 Correlation Results
The evaluation measures rank correlation between human- and machine-generated usage-similarity matrices, with Table 2 reporting results for individual words. Significant positive correlations are observed for 10 of 16 words.
- 10 out of 16 words show a significant positive correlation between human- and machine-generated usage similarity matrices.The correlations are measured using Spearman’s rank correlation and evaluated with the Mantel test.
- Table 2 reports the correlation coefficients and p-values obtained separately for each word.
- Figures 4 and 5 provide annotation instructions relevant to the human usage-similarity judgements.