Source-linked AI summary
Fitting heavy tailed distributions: the poweRlaw package
Colin S Gillespie
TL;DR
Power-law fitting is widespread, but common fitting practices can be inappropriate and many apparent power laws may also fit other heavy-tailed distributions. The paper presents poweRlaw, an R package for principled fitting, uncertainty estimation, visualization, and model comparison; examples include plausible power-law fits for Moby Dick data. Its hypothesis-testing bootstrap can be unsuitable for discrete power laws when α ≤2 because infinite means produce extremely large simulated values.
Problem
Common log-log fitting is poor, and many apparent power-law relationships could equally arise from other heavy-tailed distributions.
Method
poweRlaw provides functions for fitting distributions, estimating parameter uncertainty by bootstrap, testing power-law hypotheses, and comparing models with Vuong’s test.
Results
For Moby Dick word frequencies, the power-law hypothesis produced p = 0.678, and comparison with another model produced p = 0.682.
Takeaways & Limitations
poweRlaw gives researchers an easy-to-use, consistent interface to fit and compare heavy-tailed distributions rather than relying on subjective fitting practices.
Takeaways & Limitations
When α ≤2, bootstrap hypothesis testing for discrete power laws may be unsuitable because the first moment is infinite and extremely large values frequently occur.
Abstract
from arXiv · showhide
Over the last few years, the power law distribution has been used as the data generating mechanism in many disparate fields. However, at times the techniques used to fit the power law distribution have been inappropriate. This paper describes the poweRlaw R package, which makes fitting power laws and other heavy-tailed distributions straightforward. This package contains R functions for fitting, comparing and visualising heavy tailed distributions. Overall, it provides a principled approach to power law fitting.
1 Introduction
Power laws have been used to describe heavy-tailed phenomena across many fields, but their ubiquity and common fitting practices have been questioned. The paper introduces poweRlaw as a straightforward, principled R interface for fitting and comparing these distributions.
- Power-law claims became widespread after studies of World Wide Web hyperlinks and models of scale-free networks.
- Heavy-tailed distributions have been used to describe phenomena including biological databases, city populations, rock-fall scars, protein interactions, animal movements, word frequencies, and conflict casualties.
- Estimating a power-law exponent from a log-log plot is a poor fitting technique, motivating systematic, statistically rigorous approaches.
- poweRlaw provides functions for fitting power laws and other heavy-tailed distributions, plotting and comparing distributions, and estimating parameter uncertainty.
2 Mathematical background
The paper develops distributional and inferential foundations for fitting heavy-tailed models. It estimates the tail threshold and scaling parameter, quantifies uncertainty by bootstrap, and tests power-law plausibility or compares alternative models.
- 2.1 The power law distribution: Power-law models can be discrete or continuous, with tail behavior governed by scaling parameter α and lower cutoff xmin.
- 2.1 The power law distribution: For 2 < α ≤3, all second and higher-order moments diverge; for 3 < α ≤m + 1, all m and higher-order moments diverge.
- 2.3 Parameter uncertainty: Bootstrap resampling re-estimates xmin and α across samples to account for uncertainty in both parameters and can run in parallel.
- 2.2 Fitting heavy tailed distributions: The lower cutoff xmin is estimated by minimizing the Kolmogorov-Smirnov distance between empirical and fitted model CDFs.
- 2.4 Alternative distributions: Goodness-of-fit bootstrapping evaluates whether data could plausibly come from a power law, while Vuong’s likelihood-ratio test compares it directly with another model.
3 Example: word frequency in Moby Dick
The Moby Dick example demonstrates poweRlaw’s workflow for fitting a discrete power law: load word-frequency data, construct a distribution object, set or estimate parameters, and estimate the tail cutoff.
- The example analyzes the frequencies of 18855 words in Herman Melville’s Moby Dick; the most common word occurs 14086 times.
- The discrete power-law workflow begins by loading the moby data and creating a discrete power-law object with the displ constructor.
- The resulting S4 reference object initially sets xmin to the smallest observed x value and α to NULL.
- Users can set xmin and α directly, for example with pl_m$setXmin(5) and pl_m$setPars(2).
- For a given xmin, the scaling parameter α can be estimated by maximum likelihood or by a parameter scan, while xmin is estimated using the Kolmogorov-Smirnov approach.
3.2 Parameter uncertainty
The package uses bootstrap resampling to quantify uncertainty in fitted parameters, visualize its behavior, and assess whether a power-law hypothesis remains plausible against alternatives.
- Bootstrap inference: Bootstrap resampling re-estimates parameters across samples to quantify uncertainty, and the procedure applies to any distribution object.It can use multiple CPU cores to speed inference.
- Bootstrap inference: The xmin search can be truncated to selected values, such as 2, 4, 6, ..., 20, when the search space is large.By default, the bootstrap checks all xmin values using maximum-likelihood parameter estimates.
- Bootstrap inference: For the Moby Dick data, 5000 iterations estimate standard deviations of 1.9 for xmin and 0.02 for α.The bootstrap plots show sequential parameter means and approximate 95% confidence intervals.
- Visualising uncertainty: Histograms summarize parameter uncertainty, while a joint scatter plot highlights dependency between α and xmin.The plots provide complementary marginal and joint views of the bootstrap results.
4 Package overview
The package organizes distributions as extensible objects and provides methods for fitting, visualizing, comparing, and quantifying uncertainty in heavy-tailed models.
- Distribution objects: Distribution objects store the data set, lower cut-off xmin, parameter values, and distribution-specific internal numerical values.The internal field supports numerical procedures and is generally not user-facing.
- Distribution objects: Available distribution classes inherit either the discrete_distribution or ctn_distribution class.Table 1 lists the distributions included in the package.
- Object-oriented framework: Object-oriented design provides two stated benefits, including straightforward addition of new distributions.The passage explicitly identifies extensibility as one benefit of the framework.
- Distribution methods: Distribution-specific methods plot probability functions, while general estimate_* and bootstrap_* methods operate on distribution objects.The package supports plotting probability mass and cumulative distribution functions, parameter estimation, and bootstrap procedures.
5 Conclusion
poweRlaw addresses concerns about overenthusiastic power-law fitting by offering an easy-to-use interface for fitting and comparing truncated heavy-tailed distributions.
- Conclusion: poweRlaw aims to reduce difficulties in fitting heavy-tailed distributions by providing a consistent interface for fitting and comparing truncated distributions.The conclusion connects the package’s purpose to concerns that supposed power-law relationships can be dubious or false.