Source-linked AI summary
Statistically Significant Detection of Linguistic Change
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, Steven Skiena
TL;DR
Words and their meanings evolve over time, particularly on the Internet, creating a need to quantify linguistic shifts across temporal corpora. The paper constructs word-usage time series and applies statistically sound change-point detection, demonstrating detected shifts across books, tweets, and Amazon reviews.
Problem
The paper addresses how to quantify changes in word meaning and usage over time, especially in rapidly evolving Internet language.
Method
The approach constructs frequency, syntactic, and distributional time series for words and uses a change-point detection algorithm to assign statistical significance to detected shifts.
Results
The method detected historical semantic shifts in Google Books and recent events reflected in Tweets and Amazon Reviews, including shifts associated with storms and releases.
Takeaways & Limitations
The demonstrated capability to detect meaning shifts may help decipher ambiguity in dynamic natural-language systems and has implications for semantic search.
Takeaways & Limitations
Frequency-based metrics are vulnerable to sampling error from corpus domain and genre bias, and temporal events can increase usage without changing meaning.
Abstract
from arXiv · showhide
We propose a new computational approach for tracking and detecting statistically significant linguistic shifts in the meaning and usage of words. Such linguistic shifts are especially prevalent on the Internet, where the rapid exchange of ideas can quickly change a word's meaning. Our meta-analysis approach constructs property time series of word usage, and then uses statistically sound change point detection algorithms to identify significant linguistic shifts. We consider and analyze three approaches of increasing complexity to generate such linguistic property time series, the culmination of which uses distributional characteristics inferred from word co-occurrences. Using recently proposed deep neural language models, we first train vector representations of words for each time period. Second, we warp the vector spaces into one unified coordinate system. Finally, we construct a distance-based distributional time series for each word to track it's linguistic displacement over time. We demonstrate that our approach is scalable by tracking linguistic change across years of micro-blogging using Twitter, a decade of product reviews using a corpus of movie reviews from Amazon, and a century of written books using the Google Book-ngrams. Our analysis reveals interesting patterns of language usage change commensurate with each medium.
1. INTRODUCTION
The paper introduces a statistically significant approach to detecting how word meanings and usage shift over time, especially across rapidly changing media. It models word evolution with alternative time series and demonstrates the method across books, tweets, and online reviews.
- Motivation and problem: The paper targets statistically significant shifts in words’ meanings and usage across micro-blog posts, product reviews, and books.It specifically seeks to detect broadening and narrowing of semantic senses over a medium’s lifetime.
- Findings: The distributional method identifies a semantic transition for gay that began by 1975 and accelerated thereafter.Earlier usage associated gay with meanings such as cheerful or dapper; the example spans 1900–2005.
- Findings: For Hurricane and Sandy, frequency spikes for both words, but the distributional method detects a meaning change only for Sandy.This contrasts frequency-based activity with usage-based semantic change detection.
- Evaluation and implications: The method scales across years of Twitter posts, a decade of Amazon movie reviews, and a century of Google Books Ngram data.The authors report detecting new products, movies, books, storms, and historical semantic shifts across these media.
- Approach: Three time-series methods model word evolution using frequency, part-of-speech distributions, and word co-occurrences.The distributional method captures contextual cues from co-occurrence statistics.
- Approach: A change point detection algorithm establishes the statistical significance of word changes over time and is compatible with all three methods.The approach is designed to identify when significant changes occur, not only whether usage changes.
2. PROBLEM DEFINITION
The problem is to quantify linguistic shifts in word meaning and usage across temporal corpora. The framework represents each word with a usage time series, then asks whether a shift is significant and when it occurred.
- Problem setup: The corpus is divided into n temporal snapshots, while a shared vocabulary retains words appearing in every snapshot.This removes trivial shifts caused by words appearing or disappearing from the corpus.
- Problem setup: Each word receives a time series whose points contain statistical information extracted from successive corpus snapshots.Different construction methods use different statistical information to represent word usage.
- Detection questions: Sudden increases or decreases in a word’s time series indicate possible shifts in its usage.The framework uses these temporal changes to quantify the significance of linguistic change.
- Detection questions: The formulation asks how statistically significant a word’s usage shift is across time.This question is evaluated on the word’s constructed time series.
- Detection questions: Given that a word shifted, the framework asks when the change happened.This identifies the change point within the temporal sequence.
3. TIME SERIES CONSTRUCTION
The paper constructs word-level time series using frequency, syntactic, and distributional information, with each method capturing different aspects of linguistic evolution. Distributional modeling uses temporally trained and aligned word embeddings to track semantic displacement over time.
- 3.1 Frequency Method: Frequency time series track changes in a word’s occurrence probability across corpus snapshots, capturing sudden usage changes such as gay’s late-1980s jump.The approach uses unigram language models for each snapshot and represents frequency on a logarithmic scale.
- 3.1 Frequency Method: Frequency metrics are simple but can mistake corpus sampling bias or event-driven popularity spikes for changes in meaning.The paper illustrates this limitation with frequency increases caused by temporal events or prominent entities.
- 3.2 Syntactic Method: Syntactic time series track changes in a word’s part-of-speech distribution using divergence between distributions across snapshots.For apple, the rise of the “Proper Noun” tag in the 1980s reflects its use for Apple Computer Inc.
- 3.3 Distributional Method: Distributional time series address semantic changes that do not alter part of speech by learning word embeddings from contextual co-occurrences over time.The method learns temporal representations, aligns snapshot-specific embedding spaces, and tracks word displacement in the shared semantic space.
4. CHANGE POINT DETECTION
The method detects statistically significant linguistic changes by modeling normalized word-usage time series and testing for mean shifts at candidate change points. It combines stochastic-drift normalization, mean-shift analysis, and bootstrap-based significance estimation.
- The system determines whether a word changed significantly and estimates the corresponding change point from its usage time series.
- The method normalizes each word’s time series to account for general stochastic language drift before detecting shifts in its mean.
- The mean-shift model represents each observation as a time-varying mean plus independent zero-mean error, with changes occurring at relatively few points.
- Mean shifts compare the means of the two time-series segments on either side of a candidate time point, identifying change points through significant differences.
- Bootstrap significance testing typically uses B = 1000 permuted time-series samples to estimate p-values under a no-change null distribution.
- The estimated change point is the time point with the minimum p-value.
5. DATASETS
The study evaluates linguistic change across books, online forums, and micro-blogs using datasets spanning periods from 24 months to five centuries. The datasets include Twitter posts, Amazon movie reviews, and Google Books ngrams.
- The datasets span three modes of expression: books, online forums, and micro-blogs.
- The Google Books Ngram Corpus contains short text phrases extracted from books in eight languages over five centuries.
- The Google Books analysis uses 5-gram phrases, restricting the context window size m to 5, across 1900–2005 in 21 five-year snapshots.
- The Amazon Movie Reviews dataset contains 8 million reviews spanning August 1997 to October 2012, with analysis beginning in 2000 because earlier years contain considerably fewer reviews.
- The Twitter dataset covers 24 months from September 2011 to October 2013, with each tweet including its identifier, text, and available geolocation.
6. EXPERIMENTS
The experiments compare Frequency, Syntactic, and Distributional time-series methods for detecting linguistic shifts across historical, review, and social-media data. Distributional methods generally balance detection errors well and outperform alternatives under stronger perturbations, while each method captures different usage signals.
- 6.1 Time Series Analysis: The experiments compare Frequency, Syntactic, and Distributional methods for constructing linguistic-shift time series and detecting change points.Table 2 reports method-specific time series and p-values, while Table 3 compares estimated change points and detected senses.
- 6.1 Time Series Analysis: Frequency detects temporary popularity spikes, producing false positives for words such as her and desk that did not undergo semantic change.The authors attribute the spike in her to the rise of the feminist movement and conclude that frequency analysis is poorly suited to detecting linguistic shift.
- 6.1 Time Series Analysis: Distributional offers a good balance between false positives and false negatives without requiring linguistic resources, whereas Syntactic has low false positives but high false negatives.Syntactic also relies on linguistic taggers, which require annotated datasets and do not work well across domains.
- 6.2 Distributional Analysis: Distributional detects contextual meaning changes such as tape acquiring the sense of cassette tape in the early 1970s.The shift followed magnetic-tape introduction and mass cassette production, becoming statistically significant after the meaning evolved.
- 6.3 Cross-Domain Analysis: The methods identify shifts caused by products, media, technologies, games, applications, storms, and cultural events across books, Amazon Reviews, and Twitter.Examples include streaming, twilight, sandy, shades, candy, mystery, rally, plastic, diet, apple, windows, and bush.
- 6.4 Quantitative Evaluation: Distributional consistently outperforms Frequency across perturbation levels, and outperforms other methods when preplacement > 0.4 without language-specific resources or annotators.Syntactic outperforms Distributional only when perturbation is statistically minimal, then declines as perturbation increases.
7. RELATED WORK
The paper relates its approach to linguistic-shift research, word embeddings, change-point detection, and Internet linguistics. It distinguishes its fine-grained, statistically significant tracking of individual word shifts and emphasizes broad temporal and domain coverage with limited linguistic resources.
- Linguistic Shift: Prior studies analyzed frequent patterns, compared language across periods, or quantified broader change, whereas this work tracks individual shifts in word meaning.The authors describe this as fine-grained detection that can still quantify natural-language change as a whole.
- Word Embeddings: Earlier distributional and topic-modeling work often used only two language snapshots, omitted change-point detection, or focused on entities such as Iraq.The proposed time-series methods are presented as requiring minimal linguistic knowledge and resources across languages and domains.
- Word Embeddings: Unlike sequential embedding training, the paper warps separately trained embedding spaces after training, enabling efficient training that can be parallelized for large corpora.This design separates representation learning for each time snapshot from cross-time alignment.
- Internet Linguistics: Its datasets span different time scales, cover larger user interactions, and represent a broader sample of the web than the authors’ cited prior work.The related domains include online forums, blogs, social media, text messaging, and other Internet-influenced media.
- Change Point Detection: The work combines temporal modeling with established change-point detection and statistical-significance procedures.Its related-work context includes control charts, CUSUM, Bayesian online detection, bootstrapping, and classical change-point surveys.
8. CONCLUSIONS AND FUTURE WORK
The paper presents three time-series approaches and a computational method for statistically significant linguistic-shift detection, demonstrated across books, tweets, and Amazon Reviews. It identifies historical and recent semantic changes and points toward applications in Semantic Search and Internet Linguistics, with future work adding real-time and contextual attributes.
- Conclusions: The paper models word evolution with three time-series construction methods and designs a computational approach to detect statistically significant linguistic shifts.The conclusion summarizes the methods as applied across three datasets representing different media.
- Conclusions: Google Books Ngram analysis detects historical shifts such as gay and bitch, while Tweets and Amazon Reviews reveal recent storms, games, and book releases.The examples span the century-scale and faster-evolving media studied in the experiments.
- Conclusions: The authors state that detecting meaning shifts can help decipher ambiguity in dynamic natural languages and has implications for Semantic Search and Internet Linguistics.The conclusion frames these as implications rather than established application outcomes.
- Future Work: Future work will add real-time analysis and attributes such as geographic location and content source to study mechanisms and influential participants in meaning change.The proposed extensions target richer contextual analysis beyond the current datasets and attributes.