Source-linked AI summary
Temporal Analysis of Language through Neural Language Models
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, Slav Petrov
TL;DR
The paper asks how language change can be detected automatically over time despite historically limited corpora. It chronologically trains yearly neural language models on Google Books data, then compares word vectors to identify changed words and periods of change. The method identifies examples such as cell and gay and links cell’s detected change to 1985–2009.
Problem
Digitized historical corpora were scarce, limiting contemporary machine-learning approaches to analyzing language change across time.
Method
The authors chronologically train neural language models on yearly Google Books Ngram data and compare same-word vector similarities across years.
Results
The model identifies changed words such as cell and gay, detects subtler changes, and identifies the specific periods during which words changed.
Takeaways & Limitations
The method provides automatic identification of changed words and their periods, with cell’s detected change occurring during 1985–2009 alongside cell-phone adoption.
Takeaways & Limitations
Words whose usage decreased dramatically or ceased may retain unchanged vectors and therefore remain undetected.
Abstract
from arXiv · showhide
We provide a method for automatically detecting change in language across time through a chronologically trained neural language model. We train the model on the Google Books Ngram corpus to obtain word vector representations specific to each year, and identify words that have changed significantly from 1900 to 2009. The model identifies words such as "cell" and "gay" as having changed during that time period. The model simultaneously identifies the specific years during which such words underwent change.
1 Introduction
The paper addresses language change across time by training yearly neural language models on digitized historical text. It automatically detects changed words and identifies the periods when their usage shifted.
- Language changes through new senses, newly created words, and words falling out of use.Examples include gay, internet, and irregular verbs such as burnt.
- Scarce digitized historical corpora traditionally limited the use of data-intensive machine-learning methods for temporal language analysis.The Google Books Ngram corpus increased interest in analyzing cultural and linguistic change through digitized texts.
- The framework trains a neural language model on yearly corpora to obtain word vectors for each year from 1900 to 2009.Vectors for each subsequent year are initialized from the previous year’s vectors.
- Cosine similarity between same-word vectors across years identifies words that moved significantly in vector space.The model identifies words such as cell and gay as changed and also locates the specific periods of change.
2 Related Work
Prior work used varied computational approaches to study diachronic language change, while this paper emphasizes automatic identification of changed words and their periods of rapid change.
- Earlier studies used supervised learning, Latent Semantic Analysis, and Topics-over-Time with clustering to investigate temporal language change.These approaches predicted time periods, measured semantic change, or identified topic and cluster transitions for selected words.
- Previous work generally required researchers to manually identify words that had changed.The paper notes Gulordava and Baroni (2011) as an exception.
- This work automatically identifies changed words while also capturing yearly movement and periods of rapid change.It simultaneously identifies which words changed and when those changes occurred.
3 Neural Language Models
Neural language models represent words in a lower-dimensional vector space learned from context, and the paper uses Skip-gram vectors trained chronologically on yearly Google Books data.
- Neural Language Models: Neural language models project sparse 1-of-V word encodings into a lower-dimensional vector space through a hidden layer.The resulting vectors represent semantic properties, with semantically close words positioned near one another.
- Neural Language Models: Skip-gram learns word vectors by using each corpus word to predict a window of surrounding words.The architecture is used because it allows efficient estimation of vectors from large corpora.
- Neural Language Models: The word representations are learned in the hidden layer using stochastic gradient descent and backpropagation.Skip-gram is computationally efficient and competitive with other vector-space models when trained on the same data.
- Training: The training data samples 10 million 5-grams per year from the English fiction corpus for 1850–2009.The vocabulary is restricted to words occurring at least 10 times across the full corpus.
- Training: The model uses a window size of 4 and 200-dimensional word vectors, iterating within each year until convergence.Convergence is defined by the average angular change in word vectors between epochs; the learning rate decreases from 0.01 to 0.0001.
- Training: For each subsequent year, training is initialized with the previous year’s converged word vectors.The process is repeated chronologically from 1850 through 2009.
4 Results and Discussion
Comparing yearly word-vector cosine similarities identifies words whose usage changed between 1900 and 2009 and reveals how their neighboring words and periods of change shifted. The analysis illustrates these changes with checked, actually, cell, and gay while noting important frequency-related limits.
- Word comparisons: Cosine similarity between the same words across years identifies words whose usage changed, while function words largely appear among the least changed.The analysis lists intuitive candidates such as gay, cell, and ass alongside less obvious candidates such as checked, headed, and actually.
- Word comparisons: Examples from Google Books indicate that checked shifted from restraint-related usage in 1900 toward verification or inspection-related usage in 2009.The authors caution that this interpretation may oversimplify a highly polysemous word.
- Word comparisons: Examples suggest that actually changed in sense popularity from referring to what is true or real toward expressing wonder or surprise.The cited examples contrast 1900 and 2009 usages.
- Periods of change: Neighbor-based time-series plots compare cell and gay with period-specific neighboring words, while reference-year cosine plots distinguish change from average drift.For cell, the identified change period is 1985–2009; for gay, it agrees with the movement beginning around the 1970s.
- Limitations: The method requires words to occur often enough during the study period, so words whose usage sharply declines or stops may not be detected as changed.The paper suggests combining cosine distance with frequency as a possible remedy.
5 Conclusions and Future Work
The paper presents a chronologically trained neural language-model method that automatically identifies changed words and the periods when their usage changed. It deliberately leaves the type of linguistic change unclassified, motivating future research.
- The method uses word vectors from a chronologically trained neural language model to analyze written-language change across time.
- It automatically identifies words that have changed and the periods during which those changes occurred.
- The analysis deliberately avoids inferring whether changes are semantic, syntactic, broadening, narrowing, pejorative, or ameliorative.
- Future work could characterize change types by examining whether patterns of movement in vector space correspond to particular kinds of linguistic change.