PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-scale analysis of Zipf's law in English texts

Isabel Moreno-Sánchez, Francesc Font-Clos, Álvaro Corral

arXiv:1509.04486v1stat.APphysics.soc-ph

TL;DR

The paper addresses uncertainty about how Zipf’s law should be formulated and how well it describes individual texts. It tests three frequency-distribution versions using maximum-likelihood fitting and goodness-of-fit tests on English Project Gutenberg texts. More than 40% of the texts are compatible with a pure power law in the complementary cumulative frequency distribution at the 0.05 significance level.

  • Problem

    The validity and range of Zipf-like descriptions across individual texts are unknown, and rank-frequency and frequency-distribution formulations are not statistically equivalent for discrete data.

    Source: INTRODUCTION · pp. 1–2

  • Method

    The authors fit three discrete frequency distributions by maximum likelihood and assess fit with a discrete Kolmogorov–Smirnov test across 31,075 English texts.

    Source: INTRODUCTION · p. 2

  • Results

    More than 40% of the texts are statistically compatible with a pure power law in the complementary cumulative frequency distribution, and about twice as many texts fit it as the probability-mass power law.

    Source: CONCLUSIONS · p. 9

  • Takeaways & Limitations

    For this corpus, the complementary cumulative distribution version is the most appropriate of the tested versions of Zipf’s law.

    Source: CONCLUSIONS · p. 9

    The tested distributions are single-parameter models; more complex extensions remain to be explored and would require balancing fit against parsimony.

    Source: CONCLUSIONS · p. 8

Abstract

from arXiv · show

Despite being a paradigm of quantitative linguistics, Zipf's law for words suffers from three main problems: its formulation is ambiguous, its validity has not been tested rigorously from a statistical point of view, and it has not been confronted to a representatively large number of texts. So, we can summarize the current support of Zipf's law in texts as anecdotic. We try to solve these issues by studying three different versions of Zipf's law and fitting them to all available English texts in the Project Gutenberg database (consisting of more than 30000 texts). To do so we use state-of-the art tools in fitting and goodness-of-fit tests, carefully tailored to the peculiarities of text statistics. Remarkably, one of the three versions of Zipf's law, consisting of a pure power-law form in the complementary cumulative distribution function of word frequencies, is able to fit more than 40% of the texts in the database (at the 0.05 significance level), for the whole domain of frequencies (from 1 to the maximum value) and with only one free parameter (the exponent).

INTRODUCTION

Zipf’s law is a widely observed power-law pattern, but its statistical validity across individual texts and its range of applicability remain unknown. The paper argues that frequency distributions are better suited than rank-frequency relations for statistical inference and sets out to test three Zipf-like models at large scale.

INTRODUCTION

  • The range and precision of Zipf’s law across individual texts remain unknown despite prior evidence from large aggregated corpora at high frequencies.Earlier linguistic studies typically examined fewer than a dozen texts, while aggregated corpora combine many texts into one.Source: INTRODUCTION · p. 1
  • The frequency-based formulation counts how many word types share each frequency, distinguishing word types from their repeated tokens.It treats frequency as the random variable and relates normalized rank to its empirical complementary cumulative distribution.Source: INTRODUCTION · p. 2
  • Rank-frequency analysis poses three statistical problems: reconstructed ranks can bias estimates, ranks are assigned rather than random, and untruncated power laws cannot accommodate exponents below 1.Avoiding truncation to address these issues can discard substantial data.Source: INTRODUCTION · p. 2
  • Frequency is a well-defined random variable, so the paper uses its distribution rather than rank-frequency representation for statistical inference.The authors argue that maximum-likelihood estimation cannot be applied directly and reliably to the rank-frequency representation.Source: INTRODUCTION · p. 2

ZIPF-LIKE DISTRIBUTIONS

The paper compares three discrete frequency distributions that share a power-law tail but differ in how they define the power law and behave at small frequencies. It fixes the minimum frequency at 1 to fit the full distribution rather than only its tail.

ZIPF-LIKE DISTRIBUTIONS

  • In the first model, the power law is in the probability mass function f(n), with normalization supplied by the Hurwitz zeta function.The passage identifies this form as the normalized version of the earlier frequency-distribution expression.Source: ZIPF-LIKE DISTRIBUTIONS · p. 3
  • The second model places the power law in the survival function S(n), corresponding to a power law in the empirical rank-frequency relation.For discrete frequencies, a power law in the mass function does not imply one in the survival function, or vice versa.Source: ZIPF-LIKE DISTRIBUTIONS · pp. 2–3
  • The third model uses the frequency distribution derived by Mandelbrot when ranks are generated independently from a power law.Its probability mass function is expressed using the beta function.Source: ZIPF-LIKE DISTRIBUTIONS · p. 3
  • All three distributions are normalized discrete models on frequencies starting at a, and each has an asymptotic power-law tail with exponent β.The study fixes a = 1, covering the whole frequency distribution.Source: ZIPF-LIKE DISTRIBUTIONS · pp. 2–3
  • The models differ most at small frequencies: f2 is convex and f3 concave on a log scale, while f1 is neither.For large n, their frequency mass functions share the same power-law tail.Source: ZIPF-LIKE DISTRIBUTIONS · p. 3

METHODOLOGY AND DATA

The study fits three frequency-distribution models to individual English Project Gutenberg texts using maximum likelihood and evaluates fit with a discrete Kolmogorov–Smirnov test. It applies this procedure to 31,075 texts longer than 100 word tokens.

METHODOLOGY AND DATA

  • Each text’s frequency data were fit to three distributions by maximizing the likelihood over the shared exponent parameter β.The estimator is computed numerically; for f1 the log-likelihood uses the geometric mean of observed frequencies.Source: METHODOLOGY AND DATA · p. 3
  • Fit quality was tested with the discrete Kolmogorov–Smirnov statistic, using Monte Carlo p-values based on 100 simulations per test.Because β is estimated from the tested data, the authors use simulation to avoid positively biased theoretical p-values.Source: METHODOLOGY AND DATA · p. 3
  • The analysis treats frequency observations as independent and does not apply a Bonferroni-like correction for multiple testing.The authors state that omitting the correction makes acceptance of fits more strict.Source: METHODOLOGY AND DATA · p. 3
  • 31,075 English texts longer than 100 word tokens were analyzed after filtering and selecting actual word types.The source database contained 37,041 texts; the analyzed English texts ranged up to 4,659,068 tokens.Source: METHODOLOGY AND DATA · p. 4

RESULTS

Across the Gutenberg texts, f2 fits a larger share than f1, while f3 is a poor description. The results support these distributions for subsets of texts, not the entire database.

RESULTS

  • P-values are not uniformly distributed, and most texts have rather small p-values for all three fits.For f1 and f2, however, many texts still yield high enough p-values to avoid rejection.Source: RESULTS · p. 4
  • At p=0.05, f2 fits about 40% of texts versus 15% for f1; f3 is unsuitable for any non-negligible subset.The f2-to-f1 fit ratio remains around 2.6 across significance levels.Source: RESULTS · p. 4, Figure 2 · p. 5
  • When ML-KS selects f1 or f2, the likelihood-ratio test either supports that selection or is not significant; it never significantly favors the alternative.The likelihood-ratio test compares relative fit quality and cannot establish that either model is good in absolute terms.Source: RESULTS · pp. 6–7
  • Among texts accepted by f1 or f2, estimated β values cluster around 2 and decrease with text length; synthetic texts with fixed β do not show this trend.The authors report no theoretical explanation for the length trend.Source: RESULTS · pp. 7–8
  • f1 fits more texts only below about 13,000 tokens; f2 dominates longer texts, fitting many above 200,000 while f1 is irrelevant beyond 100,000 at p=0.05.The relative performance pattern holds at significance levels 0.05, 0.20, and 0.50.Source: RESULTS · p. 6, Figure 7 · p. 7

CONCLUSIONS

Across 31,075 Project Gutenberg texts, the authors compare three one-parameter interpretations of Zipf’s law under strict full-frequency-range testing. The complementary-cumulative power law f2 is the best-supported version in this corpus, though richer extensions remain untested.

CONCLUSIONS

  • The analysis tests three interpretations of Zipf’s law, each with a single parameter, β, the exponent of an asymptotic power law in word-frequency probability.The authors leave more complicated extensions for future exploration, noting that model selection must balance fit against parsimony.Source: CONCLUSIONS · p. 8
  • More than 40% of considered texts are statistically compatible with f2’s pure-power-law complementary cumulative distribution, making f2 the most appropriate Zipf’s-law version for the corpus under consideration.Its survival function is S(n) = 1/n^(β−1).Source: CONCLUSIONS · p. 9
  • Across significance levels, about twice as many texts are compatible with f2 as with f1, while f3 is irrelevant for this corpus.The absolute number of accepted texts changes with the significance level.Source: CONCLUSIONS · p. 9
  • The exponent β varies across texts, and β=2 is among the most frequent values.Source: CONCLUSIONS · p. 9
  • The authors speculate that f2 may fit a large fraction of generic, non-technical English texts, but do not test that claim across all possible corpora.Source: CONCLUSIONS · p. 9

APPENDIX: SIMULATION OF DISCRETE ZIPF-LIKE DISTRIBUTIONS

The appendix generates discrete f2 samples by transforming uniform random draws, then uses f2 as an auxiliary distribution for rejection sampling of f1 and f3. It also describes the rejection efficiency and a simplified acceptance calculation for f1.

APPENDIX: SIMULATION OF DISCRETE ZIPF-LIKE DISTRIBUTIONS

  • For f2, a uniform draw is transformed into a continuous power-law value x, then n=int(x) yields the desired discrete survival function S2(n)=(a/n)^(β−1).The construction applies for integer n≥a, with umax=1/a^(β−1).Source: APPENDIX: SIMULATION OF DISCRETE… · p. 9
  • The authors use f2 as the auxiliary distribution in rejection sampling for f1 and f3, accepting proposals according to a ratio-based condition.For f1, the rejection constant is C=max_{n≥a}{f1(n)/f2(n)}.Source: APPENDIX: SIMULATION OF DISCRETE… · p. 9
  • For f1, the acceptance condition can be simplified without calculating the Hurwitz-zeta function; at β=2 and a=1, C=1.2158.For f3, C is also low and approaches 2 as β→2 for a=1.Source: APPENDIX: SIMULATION OF DISCRETE… · pp. 9–10

Related papers

Loading 1509.04486v1…