Source-linked AI summary
Survey of Computational Approaches to Lexical Semantic Change
Nina Tahmasebi, Lars Borin, Adam Jatowt
TL;DR
The paper addresses the need to understand lexical semantic change and related lexical replacement as computational methods are increasingly applied to historical texts. It surveys computational approaches, including context-based, topic-modeling, and embedding methods, together with evaluation challenges and applications. The survey finds that word-embedding methods represent the state of the art, while most remain sense-agnostic and model mixtures of senses.
Problem
The field lacks a solid, extensive overview of computational research on lexical semantic change, while large digitized corpora make systematic investigation and evaluation increasingly important.
Method
The paper surveys computational approaches to diachronic conceptual change, including context-vector, dynamic topic-model, and word-embedding methods, and discusses their evaluation and applications.
Results
Word-embedding methods represent the state of the art, but most approaches are sense-agnostic and focus on mixtures of word senses expressed by a lexeme.
Takeaways & Limitations
Automatic semantic-change detection is a developed research area, but automatically generating a concise story explaining how and why a term changed remains unresolved.
Takeaways & Limitations
Evaluation is constrained by the lack of open datasets and automatic evaluation methods, while corpus representativity and the choice of temporal reference remain unresolved issues.
Abstract
from arXiv · showhide
Our languages are in constant flux driven by external factors such as cultural, societal and technological changes, as well as by only partially understood internal motivations. Words acquire new meanings and lose old senses, new words are coined or borrowed from other languages and obsolete words slide into obscurity. Understanding the characteristics of shifts in the meaning and in the use of words is useful for those who work with the content of historical texts, the interested general public, but also in and of itself. The findings from automatic lexical semantic change detection, and the models of diachronic conceptual change are currently being incorporated in approaches for measuring document across-time similarity, information retrieval from long-term document archives, the design of OCR algorithms, and so on. In recent years we have seen a surge in interest in the academic community in computational methods and tools supporting inquiry into diachronic conceptual change and lexical replacement. This article is an extract of a survey of recent computational techniques to tackle lexical semantic change currently under review. In this article we focus on diachronic conceptual change as an extension of semantic change.
1. Introduction
This survey addresses the rapid growth of computational research on lexical semantic change and lexical replacement, offering an overview of methods, concepts, evaluation, and applications. It aims to support both established researchers and newcomers to the field.
- Motivation: Computational research on lexical semantic change has surged, enabled by digitized historical documents, large-scale corpora, and growing information-access needs.These resources support testing computational approaches and quantitatively examining hypotheses about vocabulary change.
- Scope and purpose: The survey fills a lack of solid, extensive overview of the growing computational field.It complements related surveys with narrower or overlapping scopes.
- Survey coverage: The survey covers diachronic conceptual change, including word-level and sense-differentiated changes in word meanings over time.This strand connects computational linguistic research with linguistic work and large-scale studies of semantic change.
- Survey coverage: A second strand examines lexical replacement, where different words express the same meaning across time, complicating access to historical archives.Relevant information may be retrievable only under an obsolete label for an entity or phenomenon.
- Survey organization: The survey also provides background on terminology, methodological and evaluation issues, and research-oriented and downstream applications.Applications include vocabulary-change visualizations and information-access systems.
- Purpose: The authors present the survey as useful for researchers and newcomers seeking recent advances and promising research directions.They specifically mention PhD candidates as an example of the intended audience.
2. Conceptual Framework and Application
The conceptual framework distinguishes lexical variation, semantic change, and lexical replacement while treating diachronic conceptual change as an overarching view of changing lexical meanings. It also emphasizes that identifying comparable forms and senses is theoretically and computationally difficult.
- Conceptual framework: Lexical variation must be distinguished from lexical change because variation can arise from linguistic and extralinguistic factors without constituting diachronic change.The task of identifying variants is logically separate from classifying the variation as lexical change.
- Forms and variation: Observed differences in lexical form across texts from different periods may reflect synchronic variation, orthographic differences, or spelling reforms rather than diachronic causes.Written materials can mix older and newer orthographies, and variation may also reflect pronunciation, register, or context.
- Forms and variation: Over long periods, sound change can eliminate shared surface sounds between cognates, but the surveyed work mainly treats shorter-span written-text variation as spelling change.The authors contrast Polish w and Swedish i, both meaning “in,” with the shorter historical spans targeted by the survey.
- Analytical challenges: Semantic-change analysis requires identifying comparable lexical units on the form side and grouping relevant senses on the content side.These tasks involve issues such as multiword expressions, segmentation, language variation, and competing sense inventories.
- Conceptual framework: Diachronic conceptual change is proposed as a superordinate concept covering changes in a language’s set of lexical meanings.It includes new words with new senses and existing words acquiring new senses, while considering senses and word-sense allocations across the language.
- Conceptual framework: Lexical replacement concerns different words expressing the same meaning over time, alongside related computational phenomena such as named-entity change and temporal analogy.Examples include Volgograd replacing Stalingrad, foolish replacing nice for one sense, and iPod as a temporal analog of Walkman.
3. Computational Modeling of Diachronic Semantics
The surveyed approaches model diachronic conceptual change with distributional representations, including co-occurrence vectors and embeddings, and evaluate changes through similarity, neighbors, clusters, and temporal drift. Results show that performance depends on representation, corpus size, evaluation design, and whether words are modeled with single or dynamic representations.
- Representation methods: Pointwise mutual information uses Google Books Ngram 2-grams, representing each word through its other word in the pair.
- Word-level detection: 0.445 was the highest reported Pearson correlation between distributional similarity and manually judged change for words more frequent in the 90s.The frequency method reached 0.310 for words more frequent in the 60s.
- Word-level detection: 40% of reference-set words were retrieved by the distributional method, compared with 15% by the syntactic method.This evaluation was intended to capture recall of known changes.
- Word-level detection: 53.33% of the distributional method’s top 20 words were judged to have changed meaning, versus 21.66% for syntactic and 13.33% for frequency methods.The evaluation classified words as changed or unchanged, without tying judgments to change points.
- Representation comparison: 70% of SGNS’s top 10 words were judged correct, compared with 40% for SVD and 10% for PPMI.Corpus sampling affected interpretation: ENGALL’s top results were dominated by scientific terms, and PPMI’s remaining items were borderline.
- Dynamic representations: Dynamic embeddings outperformed Static-Word2Vec, Transformed-Word2Vec, and Aligned-Word2Vec on NMI and F-measures.Transformed-Word2Vec showed the largest performance drop among the baselines.
4. Computational Modeling of Diachronic Word Replacement
The survey organizes diachronic replacement into four types and reviews computational methods, especially named-entity replacements and temporal analogs. These methods support historical search and other NLP applications but often ignore word senses.
- Replacement types: Diachronic replacement includes lexical replacements, changing names for the same entity, successive instances of one type, and temporal analogs.The first type concerns synonymous lexemes, while the latter three concern entities, type instances, and cross-time similarity based on roles or functions.
- Limitations and applications: Diachronic replacement detection can support education, search engines, NLP pipelines, and reformulation of historical queries for document archives.A Hidden Markov model for query reformulation combines across-time similarity with coherence and target-period popularity, though it may require recurrent computation per query.
- Named entities: Named-entity replacement methods use association rules, hyperlink histories, burst detection, contextual co-reference analysis, or classifiers to discover temporal name variants.NEER, for example, detects periods likely to contain name changes and analyzes contexts for temporal co-references.
- Named entities: 90% precision on known periods and 93% on found periods were achieved by NEER on 16 entities with 33 names and 86 co-references.The experiments used a random forest classifier on the New York Times dataset.
- Temporal analogs: Embedding-based temporal-analog detection aligns vector spaces from different periods, but requires sufficiently large training data across domains, genres, and time spans.Hierarchical clusters with multiple transformation matrices improved mapping over a single linear transformation matrix in one study.
- Limitations and applications: Most reviewed replacement approaches are sense-agnostic, mixing senses or relying on a dominant sense rather than matching analogous terms by sense or aspect.The survey identifies sense- or topic-specific matching as a future improvement, using Walkman/iPod and iPod/PC as contrasting functional analogies.
5. Linguistic Approaches to Vocabulary Change
Linguistic approaches frame vocabulary change through lexical items, word senses, and the form–meaning relation, while distinguishing semantic change from lexical replacement and grammaticalization. The survey emphasizes cross-linguistic analysis, classification of change, and the difficulty of isolating senses and causes.
- Analytical perspectives: A word sense is the combination of a lexical item and one recognized meaning, and forms and meanings stand in a many-to-many relation through polysemy and synonymy.The examples include bank with financial-institution and sea-floor senses, and deer referring to an animal or its meat.
- Computational and linguistic approaches: Word senses are difficult to isolate, and using published lexicons as canonical gold standards leaves theoretical and methodological questions unresolved.The survey does not take a position on precisely how senses should be defined or identified.
- Core concepts: Historical linguistics studies vocabulary developments under lexical change, semantic change, grammaticalization, and lexical replacement.Lexical change is used here as the broad cover term for diachronic vocabulary change, while another usage narrows it to entering and leaving word forms.
- Core concepts: Semantic change concerns an existing form acquiring or losing a meaning, whereas lexical replacement concerns one synonymous lexeme ousting another over time.The survey gives adrenaline/epinephrine as an example of lexical replacement and describes semantic change as increasing or decreasing polysemy.
- Analytical perspectives: Linguistic analysis distinguishes semasiological studies of forms and their meanings from onomasiological studies of how particular meanings are expressed.Semantic-change studies generally take the semasiological perspective, while other lexical-change work generally focuses on onomasiology.
- Change processes: Grammaticalization is a semantic-change pathway in which content words become function words and ultimately bound grammatical morphemes.The French preposition chez is presented as developing from the Latin noun casa.
- Change processes: Linguists classify observed changes into cross-linguistically valid types and investigate material, cognitive, community, contact, and speaker-population factors that may influence them.The survey also connects this work with lexical typology, which studies how languages categorize domains through lexical items.
- Computational and linguistic approaches: Computational work could offer historical linguistics a novel direction, but the survey says it must engage with historical-linguistic theory, methods, and terminology.The authors specifically note that awareness of the field's state of the art is sometimes lacking in the surveyed work.
6. Methodological Issues and Evaluation
The survey emphasizes that evaluating diachronic conceptual change requires explicit grounding in both corpus evidence and the outside world, while dataset representativity and timing remain difficult to establish. It recommends evaluation procedures that address frequency bias, source grounding, and cross-linguistic and genre limitations.
- Evaluation and hypothesis testing: No open datasets or automatic evaluation methods currently provide a standard regime for diachronic conceptual change detection.Existing studies use varied techniques, datasets, dimensions, and often manual evaluation; WordNet lacks information about when meanings changed.
- Applicability and representativity: A valid evaluation must determine whether computational representations capture a word’s meaning or all of its senses completely and correctly.This includes evaluating representations produced by clustering, topic modeling, or nearest-neighbor word spaces.
- Applicability and representativity: Language coverage and corpus composition constrain generalization because datasets may overrepresent English, written language, particular genres, regions, or social strata.The survey calls for attention to representativity relative to the research question and for experiments across more languages.
- Applicability and representativity: Corpus-based variation cannot establish diachronic conceptual change unless other relevant variables are controlled across time slices.The survey notes that changing OCR quality, tokenization, and related factors can generate misleading patterns.
- Factors involved in evaluation: The evaluation target for change timing may be the outside world or the investigated dataset, and gradual shifts make exact timing difficult to define.Dataset delays can penalize systems evaluated against invention dates; for example, the computing-device sense of computer reached a frequency threshold in 1934 in German, 1943 in American English, and 1953 in British English.
- Datasets and testsets: Google Books Ngrams provide very large multilingual time series, but earlier periods are sparse, OCR errors may be frequent, and scientific publications increasingly dominate the data.COHA offers more stable genre composition but omits many rare words, limiting its usefulness to relatively common terms.
7. Applications and Online Systems
The survey describes online systems and visual analytics that complement automatic diachronic conceptual change detection with interactive exploration of frequencies, contexts, meanings, and semantic trajectories. These systems range from general corpus interfaces and frequency plots to specialized visualizations and multi-view analytical frameworks, although exact sense-change timing remains difficult to determine.
- Systems supporting analysis: Interactive visualization systems help verify automatic results and support manual investigation of diachronic conceptual change.They vary in interactivity, querying freedom, and support for multidimensional analysis.
- Systems supporting analysis: Google’s definition queries combine a standard definition with word origin and frequency-over-time information, leaving users to infer meaning change.The interface shows counts over time but does not itself explain semantic change.
- Corpus interfaces: The Google Books Ngram Viewer supports temporal frequency comparisons, part-of-speech searches, composition operators, and inflection-oriented queries.It is widely used in digital humanities research and is based on Google Books Ngrams datasets.
- Corpus interfaces: BYU corpus interfaces provide frequency plots, decade comparisons, keyword-in-context examples, and collocate listings without requiring code.These functions support direct inspection of word usage at different time points.
- Visual representations: Visual approaches represent semantic change through time-dependent plots, animations, matrices, heatmaps, word clouds, and trajectories in reduced vector spaces.Examples include animated scatterplots, dictionary-edition matrices, and 2D projections of word meanings across years or decades.
- Visual representations: Specialized systems such as ShiCo construct time-specific semantic spaces with word embeddings to visualize shifting concepts in Dutch.Its model generates and aggregates information across semantic spaces typically built for successive time units.
- Analytical frameworks: Jatowt and Duh’s framework combines self-similarity plots, decade heatmaps, sentiment analysis, comparative word analysis, and context listings across Google Books Ngrams and COHA.The resulting online system supports multiple viewpoints, including context comparison, term clouds, term trees, and contrastive word analysis.
- Limitations: Exact sense-change timing is often impossible to determine precisely, so visual and interactive approaches can accommodate competing interpretations.The survey expects such approaches to become more available as sense tracking remains inherently complex.
8. Summary, Conclusions and Research Directions
The survey finds rapid growth in computational lexical semantic change methods, while highlighting unresolved limitations in sense handling, data coverage, evaluation, explanation, and multilingual evidence. It recommends stronger controls, clearer justification, richer explanations, and broader linguistic coverage.
- Main observations: Embedding-based methods represent the state of the art, but most are sense-agnostic and model mixtures of a lexeme’s senses.Methods range from counting and generative approaches to neural word embeddings.
- Main observations: Low-frequency words remain difficult to analyze reliably, while dynamic embeddings may offer a more suitable alternative for small datasets.Sense-differentiated embeddings are likely to require even more data.
- Main observations: Common-vocabulary and spelling-normalization practices exclude words absent from all periods or represented by historical spelling variants.The survey calls for combining spelling and sense variation when discovering and describing language change.
- Main observations: Detecting diachronic sense change does not by itself identify its exact timing, because thresholds depend on historically limited and skewed textual evidence.Historical texts often represent only a small sample of the wider language community.
- Main observations: Evaluation remains difficult because studies use different datasets, test words, preprocessing, and metrics, with no established standards for semantic change detection.The same comparability problem also affects temporal analog detection.
- Research directions: The field needs automatically generated evidence and explanations, user-friendly visualizations, controls, and studies across representative, diverse languages.The recommendations include justifying results, using time-stable words or control datasets, and developing automatic word stories.
Biographies of Authors
The authors are researchers and academics working across natural language processing, computational and historical linguistics, language technology, text mining, and information retrieval. They are affiliated with universities and language-research infrastructures in Sweden and Japan.
- Authors: Nina Tahmasebi researches automatic diachronic language change detection, especially semantic change, alongside information extraction and change detection.She is a Natural Language Processing researcher at the University of Gothenburg and leads a project on computational lexical semantic change detection.
- Authors: Lars Borin is a University of Gothenburg professor whose interests include historical, areal, and typological linguistics and computational lexical semantics.He also co-directs Språkbanken and coordinates Swedish activities in the CLARIN ERIC infrastructure.
- Authors: Adam Jatowt is an associate professor at Kyoto University specializing in text mining, temporal information retrieval, web archive search, and information comprehensibility.His doctorate is in Information Science from the University of Tokyo.
- Authors: All three authors co-organized the 1st International Workshop on Computational Approaches to Historical Language Change alongside ACL2019.