Source-linked AI summary
Languages cool as they expand: Allometric scaling and the decreasing need for new words
Alexander M. Petersen, Joel N. Tenenbaum, Shlomo Havlin, H. Eugene Stanley, Matjaz Perc
TL;DR
The paper examines how word-frequency distributions, vocabulary growth, and linguistic change behave as written languages expand. Using massive multilingual book corpora, it finds distinct scaling regimes, decreasing marginal need for new words, and a dynamical cooling pattern in language growth.
Problem
The paper investigates how vocabulary and word-use fluctuations scale with the expanding corpus of written language.
Method
The authors analyze Google Books word frequencies across languages, define relative word use and fluctuation measures, and vary word-occurrence cutoffs to test scaling relations.
Results
Common words follow Zipf scaling while rarer words form a less-steep regime; vocabulary growth has b < 1, and fluctuation scaling yields 0 < β < 1/2.
Takeaways & Limitations
Languages need progressively fewer new words as they expand, while their word-use fluctuations decline, establishing a dynamical regularity distinct from Zipf’s and Heaps’ laws.
Abstract
from arXiv · showhide
We analyze the occurrence frequencies of over 15 million words recorded in millions of books published during the past two centuries in seven different languages. For all languages and chronological subsets of the data we confirm that two scaling regimes characterize the word frequency distributions, with only the more common words obeying the classic Zipf law. Using corpora of unprecedented size, we test the allometric scaling relation between the corpus size and the vocabulary size of growing languages to demonstrate a decreasing marginal need for new words, a feature that is likely related to the underlying correlations between words. We calculate the annual growth fluctuations of word use which has a decreasing trend as the corpus size increases, indicating a slowdown in linguistic evolution following language expansion. This "cooling pattern" forms the basis of a third statistical regularity, which unlike the Zipf and the Heaps law, is dynamical in nature.
Results
Across large multilingual book corpora, written language exhibits distinct scaling regimes and systematic growth patterns. Common words follow Zipf-like behavior, while vocabulary expansion shows decreasing marginal need for additional words and declining use fluctuations as corpora grow.
- The Zipf law and the two scaling regimes: 11 datasets spanning millions of books show a two-regime word-frequency distribution, with Zipf scaling for common words and a distinct regime for rare words.The rare-word regime includes technical terms, new words, numbers, spelling variants, and OCR errors.
- The Zipf law and the two scaling regimes: For the kernel lexicon, ζ ≈1 and α+ ≈2, whereas the unlimited lexicon has α−≈1.7 and does not obey the Zipf law.The crossover occurs around f× ≈10−5, separating common and rare usage regimes.
- Allometric scaling: Pruning rare words raises b from approximately 0.5 toward unity, showing that core words are structurally integrated while rare words alter the observed scaling relation.The relation b(Uc) ≈1/ζ is confirmed only after extremely rare unlimited-lexicon words are excluded.
- Allometric scaling: Vocabulary size follows sublinear Heaps scaling, with b < 1, indicating a decreasing marginal need for new words as corpus size increases.The marginal relation ∂Nw/∂Nu ∼ (Nu)b−1 decreases monotonically for b < 1.
- Corpora size and word-use fluctuations: From 1800–2008, literary productivity and vocabulary size generally increased, apart from periods during the two World Wars.Nu denotes total word uses, while Nw denotes distinct words digitized from books.
- Corpora size and word-use fluctuations: Word-use fluctuation decreases as corpora expand, with language-dependent β ≈0.08–0.35 and β <1/2 relative to the Yule-Simon prediction.Pruning rare words raises β from values near 0 to values below 1/2, while residual spillovers remain in the kernel lexicon.
Discussion
The discussion links language expansion to two complementary patterns: a decreasing marginal need for new words and declining fluctuations in word use as corpora grow. Together, these findings characterize language growth as continuing but dynamically cooling, while rare-word emergence counteracts expansion-driven cooling.
- Word-frequency regimes: The two-regime frequency distribution is robust across languages and data subsets, with common kernel-lexicon words following Zipf scaling and rarer unlimited-lexicon words following a less-steep regime.The paper attributes the distinction partly to specialized words, new words, numbers, spelling variants, and OCR errors in the unlimited lexicon.
- Vocabulary expansion: The analysis validates Heaps-law scaling and conditionally confirms ζ ≈1/b only when extremely rare unlimited-lexicon words are excluded.The kernel lexicon supplies a stable ζ ≈1 comparison, whereas including the rarest words prevents unconditional confirmation of the theoretical relation.
- Vocabulary expansion: Vocabulary growth exhibits economies of scale, with b < 1 indicating an increasing marginal return for new words or a decreasing marginal need for lexical expansion.The authors relate this pattern to increasing combinations and complexities among words as the vocabulary expands, while retaining dependencies between new and existing words.
- Language cooling: Word-use growth fluctuations decrease as corpus size increases, producing a power-law cooling pattern and a dynamical regularity distinct from the static Zipf and Heaps laws.The paper bounds the size-variance exponent at 0 < β < 1/2 and compares it with Gibrat and Yule-Simon predictions.
- Language cooling: New-word emergence behaves like condensation that tends to offset the cooling brought by corpus expansion, while high literary productivity further lowers growth-rate fluctuations.The discussion also connects lower fluctuations with more stable rankings of top words and phrases.