Source-linked AI summary
Powerlaw: a Python package for analysis of heavy-tailed distributions
Jeff Alstott, Ed Bullmore, Dietmar Plenz
TL;DR
Power-law fitting methods require substantial programming and statistical expertise, while empirical heavy-tailed data may also be explained by competing distributions. powerlaw packages fitting, comparison, visualization, and distribution options into an extensible Python system; its examples show only moderate support for a power law in one comparison, and indistinguishable fit quality from a lognormal for word frequencies.
Problem
Appropriate power-law fitting requires significant programming and statistical insight, and power laws can be difficult to distinguish from other heavy-tailed distributions such as lognormals.
Method
powerlaw integrates Fit and Distribution objects with multiple distribution families, fitting options, comparison methods, and extensible code for analyzing heavy-tailed data.
Results
Neither distribution is significantly stronger (p > .05), providing only moderate support for a power law without ruling out exponential truncation.
Takeaways & Limitations
Choosing among heavy-tailed distributions should incorporate domain-specific generative mechanisms and, when statistical tests cannot distinguish candidates, experiments targeting those mechanisms.
Takeaways & Limitations
When candidate distributions remain statistically indistinguishable, fitting alone cannot identify the underlying generative mechanism.
Abstract
from arXiv · showhide
Power laws are theoretically interesting probability distributions that are also frequently used to describe empirical data. In recent years effective statistical methods for fitting power laws have been developed, but appropriate use of these techniques requires significant programming and statistical insight. In order to greatly decrease the barriers to using good statistical methods for fitting power law distributions, we developed the powerlaw Python package. This software package provides easy commands for basic fitting and statistical analysis of distributions. Notably, it also seeks to support a variety of user needs by being exhaustive in the options available to the user. The source code is publicly available and easily extensible.
Basic Methods
powerlaw supports visualization and fitting of heavy-tailed data by providing PDF, CDF, and CCDF tools, selecting scaling ranges, and handling discrete distributions. Its fitting workflow identifies xmin by minimizing the Kolmogorov-Smirnov distance and can account for xmax constraints.
- Visualization: powerlaw provides functions and plotting commands for PDFs, CDFs, and CCDFs, including theoretical distributions within a Fit object.Fit objects expose pdf, cdf, and ccdf calculations and plotting methods, while Distribution objects provide theoretical curves.
- Visualization: Logarithmic bins improve PDF visualization of sparse heavy-tailed data by increasing tail observations and normalizing for wider bins.CCDFs avoid binning and are often preferred, although PDFs can better reveal tail peaks and scaling under an upper bound.
- Scaling range: The fitting workflow selects xmin by testing each unique data value and choosing the fit with the smallest Kolmogorov-Smirnov distance D.Users can also restrict the xmin search to a specified range.
- Scaling range: With xmax, CDFs and CCDFs bend downward near the upper limit, whereas PDFs remain straight, making PDFs preferable for visualizing scaling.Fit calculations and plots use data above xmin and below xmax by default when an upper limit is supplied.
- Continuous vs. discrete data: Datasets are continuous by default, but powerlaw supports properly normalized discrete distributions when data are integer-valued.Discrete calculations can be slower than continuous calculations.
Comparing Candidate Distributions
powerlaw supports comparing power-law fits with multiple candidate distributions, emphasizing that goodness of fit and domain-specific generative mechanisms are necessary for interpretation. Examples show that power laws may receive only moderate support when alternatives such as exponential truncation or lognormal fits remain plausible.
- Comparing Candidate Distributions: The Fit object exposes fitted parameters and goodness-of-fit analyses for multiple candidate distributions.Distributions can be evaluated individually or compared using bootstrapping, Kolmogorov-Smirnov tests, and loglikelihood ratios.
- Comparing Candidate Distributions: Comparative tests ask whether a power law is the best available description, rather than merely whether it can generate the observed data.Loglikelihood-ratio comparisons are described as faster than bootstrapping for this purpose.
- Comparing Candidate Distributions: The exponential distribution is the minimum alternative because a heavy-tailed distribution is typically defined as not exponentially bounded.A failure to outperform the exponential leaves little basis for calling data heavy-tailed, let alone power-law distributed.
- Comparing Candidate Distributions: powerlaw supports alternatives including exponentially truncated power laws, lognormals, and stretched exponentials, but considering unlimited candidates risks overfitting.Candidate selection should be informed by plausible domain-specific generative mechanisms.
- Comparing Candidate Distributions: For the C. elegans neuron data, the power law beats the exponential (R=16.384, p=0.024), but neither it nor the truncated power law is significantly stronger (p > .05).The authors therefore report only moderate support for a power law and do not rule out exponential truncation.
- Comparing Candidate Distributions: Word-frequency data are equally well fit by power-law and lognormal distributions (p > .05), illustrating why generative mechanisms matter.The accompanying figure compares complementary cumulative distribution functions for the data and both fitted distributions.
Creating Simulated Data
powerlaw can generate simulated observations from fitted or manually created distribution objects and refit those data to validate fitting accuracy. Simulated power-law data are used to validate estimates of α and xmin across varied parameter values.
- Creating Simulated Data: Distribution objects generate random data points through generate_random, either directly or through a Fit object.This supports simulation from theoretical distributions for tasks such as modeling.
- Creating Simulated Data: Simulated data can be refit to validate the accuracy of powerlaw’s fitting software.The validation procedure assesses whether fitting recovers the parameters used to generate the data.
- Creating Simulated Data: For one simulated fit, powerlaw estimates xmin=5.30 and α=2.50, while Figure S1 validates α and xmin across varied parameter values.The reported validation concerns the fitting of both parameters on simulated power-law data.
Advanced Considerations
powerlaw supports discrete-distribution approximations, nested likelihood comparisons, parameter constraints, and alternative distance metrics, while exposing trade-offs in speed, accuracy, and xmin selection.
- Discrete distributions: Discrete power-law fitting can use an analytic α estimate, accurate to about 1% or better when xmin ≥6 and no xmax is used.
- Discrete distributions: Fast discrete power-law random-data estimation has error above 8% at xmin=1, below 1% at xmin=5, and below .2% at xmin=10.
- Distribution comparisons: Nested-distribution likelihood comparisons require a particular p-value calculation, which powerlaw detects automatically when distribution names identify nesting.
- Parameter constraints: Parameter ranges can constrain xmin selection or restrict fits at each xmin, including custom functions such as σ/α < .05.
- xmin selection: Multiple local minima in D may represent alternative fits, and parameter requirements can exclude the lowest-D fit, making domain-specific judgment necessary.
- Distance metrics: The default Kolmogorov-Smirnov distance can be replaced by Kuiper or Anderson-Darling distances, though Kuiper performs similarly and Anderson-Darling can remove too much tail data.
The powerlaw Software
powerlaw is distributed as open Python source and installers, with dependencies on widely used scientific packages and mpmath for selected calculations.
- Source code and Windows installers are available through PyPI, and the package can be installed with pip.
- Source code is also available on GitHub and Google Code.
- The package requires NumPy, SciPy, matplotlib, and mpmath, with mpmath needed only for selected gamma-function calculations.
The Utility and Future of powerlaw
powerlaw integrates heavy-tailed-distribution analysis in a maintainable, extensible system that supports broad distribution coverage and simpler complete workflows.
- The integrated Fit and Distribution objects let users perform the full analysis pipeline with few lines of code.
- The package addresses software limitations described for existing tools, including absent integration and limited maintenance or expansion support.
- powerlaw supports numerous distribution families and subtypes, including discrete and continuous variants, without requiring special access code.
- Its modular architecture makes maintenance, user patches, and adding custom distributions straightforward.