Source-linked AI summary
Diachronic word embeddings and semantic shifts: a survey
Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, Erik Velldal
TL;DR
Diachronic semantic-shift research has grown rapidly but remains fragmented in terminology and shared practices. This survey synthesizes its concepts, embedding-based methods, comparison axes, applications, and challenges, concluding that the field needs stronger evaluation resources and broader methodological cohesion.
Problem
Research on diachronic semantic shifts has expanded across communities but lacks standardized terminology, cohesion, and shared practices.
Method
The paper surveys semantic-shift concepts, diachronic embedding methods, comparison axes, applications, and open challenges.
Results
The survey identifies prediction-based embeddings as dominant while reviewing findings on semantic-change laws and temporal semantic relations.
Takeaways & Limitations
The field has produced significant discoveries but remains relatively new and fragmented, with dedicated venues and shared tasks still lacking.
Takeaways & Limitations
Robust gold-standard test sets for semantic shifts should be created, and algorithms must work on the small datasets common in historical linguistics and digital humanities.
Abstract
from arXiv · showhide
Recent years have witnessed a surge of publications aimed at tracing temporal changes in lexical semantics using distributional methods, particularly prediction-based word embedding models. However, this vein of research lacks the cohesion, common terminology and shared practices of more established areas of natural language processing. In this paper, we survey the current state of academic research related to diachronic word embeddings and semantic shifts detection. We start with discussing the notion of semantic shifts, and then continue with an overview of the existing methods for tracing such time-related shifts with word embedding models. We propose several axes along which these methods can be compared, and outline the main challenges before this emerging subfield of NLP, as well as prospects and possible applications.
1 Introduction
Diachronic semantic-shift research has expanded through distributional word embeddings but remains heterogeneous in terminology and practices. The survey reviews its concepts, methods, results, applications, and open challenges.
- Large corpora and computational semantics have enabled data-driven research on temporal semantic shifts, with word embeddings becoming a widely used representation.
- The field spans NLP, information retrieval, and political science, producing terminology that remains far from standardized.
- The survey restricts coverage to distributional word embedding models representing lexical meaning with dense vectors from co-occurrence data.
- It organizes prior work around semantic-shift concepts, detection methods, statistical laws, temporal semantic relations, applications, and open challenges.
2 The concept of semantic shifts
Semantic shifts are temporal changes in lexical meaning reflected in changing word contexts and driven by linguistic, psychological, sociocultural, or cultural factors. These corpus patterns motivate automatic, data-driven detection.
- Words can undergo core meaning changes or subtler shifts in cultural associations as language and society change over time.
- Traditional semantic-shift research documented and categorized types such as narrowing and broadening.
- Semantic shifts can arise from linguistic, psychological, sociocultural, or cultural and encyclopedic causes.
- Large corpora support usage-based analysis because changes in collocational patterns can reflect changes in word meaning.
- The survey treats semantic-shift detection as locating words whose meanings differ across time periods or changed most, while also tracing trends and relations.
3 Tracing semantic shifts distributionally
Distributional semantic-shift research varies in data granularity, evaluation strategy, representation, and cross-time comparison. Prediction-based embeddings now dominate, while frequency methods are limited because semantic change need not alter frequency.
- Diachronic data: Researchers choose temporal granularity according to the phenomenon: shorter spans suit sociocultural shifts, whereas longer spans suit linguistically motivated shifts.
- Test sets: Diachronic corpora serve both as training data and as evaluation sources, but human-annotated ranked gold standards are difficult to obtain.
- Test sets: Synthetic evaluation tasks and cross-time alignment offer alternatives, yet synthetic words limit measurement of performance on real semantic-shift data.
- Methodology: Frequency and collocation methods preceded embeddings, but semantic shifts are not always accompanied by direct or substantial frequency changes.
- Methodology: Distributional models compress co-occurrence information into lexical vectors, and Kulkarni et al. report greater precision and explanatory power than frequency-based methods.
- Methodology: The field progressed from Random Indexing, count-based models, and graph methods toward dense SVD and prediction-based models such as SGNS.
- Methodology: Dynamic topic models remain another distributional approach, but SGNS, CBOW, and GloVe are described as dominant representations for diachronic semantic-shift analysis.
4 Laws of semantic change
Diachronic embedding studies have proposed several regularities describing how semantic change relates to word frequency, polysemy, prototypicality, and temporal dynamics. Later replication and control experiments challenge some of these laws as artifacts of representation choices and frequency effects.
- The proposed law of prototypicality links greater semantic change to greater distance from a cluster centroid.
- Eger and Mehler proposed that word vectors are linear combinations of neighbors in previous time periods.
- They also proposed that word meanings decay linearly over time, measured through a word’s similarity to itself.
- Frequent words change more slowly, while polysemous words controlled for frequency change more quickly, forming the laws of conformity and innovation.
- The law of conformity was also reported across non-temporal viewpoints, including texts produced by different political actors or genres.
- Replication and control experiments found that correlations with frequency, polysemy, and prototypicality also appeared in control conditions, questioning their validity.The authors attribute these spurious effects to representation type and conclude that the existence of semantic-change laws remains open.
5 Diachronic semantic relations
Diachronic embeddings can model semantic relations across time, extending analogy-style queries to identify corresponding words, temporal analogies, and changing word relatedness. These methods align embeddings from different periods or compare their cross-period relations to support temporal retrieval and query expansion.
- Embedding analogies motivate studying whether semantic relationships change over time by tracing the diachronic development of distributional models.The analogy task captures relations such as man–woman and king–queen through operations on word vectors.
- Temporal correspondence identifies the word in one period corresponding to a query term from another, such as iPod in the 2000s and Walkman in the 1980s.The task is proposed to improve information retrieval from document collections spanning substantial time periods.
- Temporal word analogies represent relations such as “word w1 at time period tα is like word w2 at time period tβ.”Models trained on different periods are aligned with linear transformations, after which the closest vector in the target period supplies the answer.
- Diachronic embeddings can identify when two words were maximally related, supporting more efficient query expansion in general-purpose search engines.
6 Applications
Applications of diachronic word embeddings fall into linguistic studies of how and why meanings shift and event-detection systems that mine texts for actionable signals. Longer spans suit gradual linguistic change, while shorter spans target real-world events and potential operational uses.
- Linguistic applications study the causes and dynamics of semantic shifts using corpora that generally span longer periods.Examples include tracking word or sentiment drift, identifying epoch breakpoints, studying semantic-change laws, and finding words with similar meaning trajectories.
- Event-detection applications mine shorter-span text collections for cultural semantic shifts associated with real-world events.Examples include temporal information retrieval, civil-turmoil prediction, and tracking entity popularity through word-vector norms.
- These event-oriented systems could improve user experience in production systems or support governmental policy-making.
- The survey anticipates more applications based on real-time analysis of large-scale news streams, where semantic shifts may reveal information about processes in the world.It emphasizes making such information reliable and practically useful.
7 Open challenges
The field faces methodological and evaluation challenges, including limited language coverage, small-data settings, weak gold standards, and insufficiently detailed shift analysis. Researchers also call for formal models, shared venues, and methods that classify shifts, identify causes, and account for changing sense weights.
- Methods should expand beyond English, with their cross-language validity demonstrated empirically.
- Algorithms must work on small datasets common in historical linguistics and digital humanities.
- Robust gold-standard test sets covering different semantic-shift types are needed for evaluation.
- Diachronic embeddings need rigorous formal mathematical models, including approaches based on joint learning across time spans.
- Future analyses should classify shift types, identify linguistic or extra-linguistic causes, and quantify changing sense weights.Most surveyed algorithms are not sense-aware, despite polysemy making sense handling central to shift detection.
- The community needs dedicated workshops, special issues, or shared tasks because existing evaluation tasks do not directly assess changing word meanings.
8 Summary
The survey outlines research on computational semantic-shift detection with diachronic word embeddings, covering linguistic foundations, data sources, and distributional modeling approaches. It concludes that this young field has made significant discoveries but remains fragmented and needs stronger coordination.
- The survey covers semantic-shift linguistics, diachronic data sources, and distributional models from frequentist methods to prediction-based embeddings.
- Despite recent discoveries and academic interchange, the emerging field remains fragmented and lacks dedicated workshops, special issues, or shared tasks.