Source-linked AI summary
ppmlhdfe: Fast Poisson Estimation with High-Dimensional Fixed Effects
Sergio Correia, Paulo Guimarães, Thomas Zylkin
TL;DR
Researchers often use log-linear models because multiple high-dimensional fixed effects are easier to handle, despite PPML’s suitability for nonnegative outcomes with zeros and heteroskedasticity. The paper introduces ppmlhdfe, a fast Stata command using modified HDFE-IRLS procedures, robust maximum-likelihood existence checks, and reghdfe-compatible functionality. In practice, its innovations can reduce calls to reghdfe by 50% or more.
Problem
Multiple high-dimensional fixed effects make PPML harder to estimate, encouraging continued use of log-linear regressions even where PPML is better justified.
Method
ppmlhdfe combines within-transformation, weighted least squares, modified IRLS, acceleration techniques, and robust checks for maximum-likelihood existence.
Results
The acceleration innovations can reduce total calls to reghdfe by 50% or more, while ppmlhdfe supports PPML estimation with multiple and interacted HDFE.
Takeaways & Limitations
ppmlhdfe brings fast, reliable PPML estimation with multiple HDFE closer to the ease and functionality of linear HDFE estimation.
Takeaways & Limitations
The illustrative example uses convergence of one coefficient as its stopping criterion, whereas the complete algorithm requires full deviance convergence.
Abstract
from arXiv · showhide
In this paper we present ppmlhdfe, a new Stata command for estimation of (pseudo) Poisson regression models with multiple high-dimensional fixed effects (HDFE). Estimation is implemented using a modified version of the iteratively reweighted least-squares (IRLS) algorithm that allows for fast estimation in the presence of HDFE. Because the code is built around the reghdfe package, it has similar syntax, supports many of the same functionalities, and benefits from reghdfe's fast convergence properties for computing high-dimensional least squares problems. Performance is further enhanced by some new techniques we introduce for accelerating HDFE-IRLS estimation specifically. ppmlhdfe also implements a novel and more robust approach to check for the existence of (pseudo) maximum likelihood estimates.
1 Introduction
PPML is a robust alternative to log-linear regression for nonnegative data, especially with zeros and heteroskedasticity, but multiple HDFE have made log-linear models easier to estimate. The paper introduces ppmlhdfe to provide similarly easy, fast PPML estimation with multiple HDFE while checking whether maximum likelihood solutions exist.
- Motivation: PPML requires correct specification of the conditional mean rather than a distributional assumption, allowing application beyond count data to any nonnegative dependent variable.This also makes PPML suitable when the dependent variable is nonnegative but not naturally count-valued.
- Motivation: PPML naturally accommodates zero-valued outcomes, unlike log-linear regression.
- Motivation: Under heteroskedasticity, OLS estimation of log-linearized models is inconsistent, so robust standard errors do not restore valid inference.
- Motivation: PPML is presented as a safe choice for nonnegative data with many zeros when researchers want minimal distributional assumptions.The paper highlights granular applications including expenditures, citations, sales, visits, credit, auctions, and commuting.
- Problem: Researchers still often use log-linear regressions because controlling for multiple high-dimensional fixed effects has been easier in linear models through tools such as reghdfe.
- Contribution: ppmlhdfe enables fast PPML estimation with multiple HDFE, offers functionality comparable to reghdfe, checks maximum likelihood existence, and adds acceleration techniques.The authors state that PPML with HDFE can be implemented with almost the same ease as linear regression with HDFE.
2 Stata Commands for Estimation of Models with HDFE
The Stata HDFE ecosystem progressed from linear-regression tools toward nonlinear PPML commands, with ppmlhdfe extending functionality to multiple and interacted fixed effects. Its within-transformation approach avoids repeating some costly computations, producing substantial speed gains relative to existing approaches.
- Existing Stata commands: Early Stata HDFE commands solved linear least-squares systems with iterative conjugate-gradient methods, while reghdfe later improved convergence and supported multiple HDFE and interactions.
- Existing Stata commands: reghdfe also added instrumental variables, multiple variance specifications, weights, and standard post-estimation tools for linear models with HDFE.
- Existing nonlinear commands: Earlier nonlinear extensions used IRLS or related algorithms around reghdfe, including poi2hdfe for two HDFE and ppmlsg for structural-gravity models with three fixed-effect sets.
- ppmlhdfe: ppmlhdfe estimates the same models as poi2hdfe and ppmlsg while also supporting multiple or interacted fixed effects and heterogeneous slopes.
- Related R packages: R packages including alpaca, FENmlm, and glmhdfe also estimate Poisson regressions with multiple fixed-effect levels using different combinations of within-transformation and nonlinear updating.
- Algorithmic distinction: The within-transformation approach avoids completely recomputing transformed data from scratch at every iteration, enabling significant speed gains and easier handling of heterogeneous slopes.
3 Estimation Approach
The estimation procedure recasts Poisson maximum-likelihood updates as iteratively reweighted least-squares problems and uses within-transformation to handle high-dimensional fixed effects efficiently. ppmlhdfe adds acceleration techniques that reduce repeated HDFE computations while addressing nonexistence of Poisson maximum-likelihood estimates through separation detection.
- IRLS algorithm: GLM coefficient estimates are obtained by solving pseudo-likelihood first-order conditions with Gauss-Newton updates, equivalent to recursive weighted least squares known as IRLS.The update uses a design matrix, iteration-specific weights, and a transformed dependent variable.
- HDFE estimation: HDFE-IRLS avoids directly forming the impractically large weighted cross-product matrix by applying the Frisch-Waugh-Lovell theorem to estimate only non-fixed-effect coefficients.Weighted within-transformed covariates and working outcomes yield residuals equivalent to those from the full regression.
- HDFE estimation: The resulting weighted least-squares residuals support the usual variance-covariance estimation, including heteroskedasticity and clustering adjustments.Once the non-fixed-effect coefficients converge, the weighted least-squares variance-covariance matrix is also valid for those estimates.
- Acceleration: ppmlhdfe accelerates HDFE-IRLS by reusing partialled-out variables, embedding reghdfe routines, and tightening inner-loop convergence criteria near convergence.The implementation progressively updates within-transformed variables and avoids repeating computations that remain unchanged across iterations.
- Acceleration: 50% or more reductions in reghdfe calls can result from these innovations, producing substantial computational speed gains.The paper reports this as a general practical reduction in calls rather than as a fixed runtime improvement.
- Existence of estimates: ppmlhdfe detects separated observations because Poisson maximum-likelihood estimates may not exist, then drops separated observations and subsequently removes regressors that become collinear.The default implementation uses the ir method alongside three additional separation-detection methods.
4 The ppmlhdfe Command
The ppmlhdfe command provides a reghdfe-like Stata interface for Poisson models with nonnegative outcomes and multiple absorbed fixed effects. Its options cover model specification, fixed-effect handling, convergence, separation checks, output, and stored estimation results.
- Syntax and requirements: ppmlhdfe requires the latest versions of ftools and reghdfe and uses syntax similar to reghdfe.The command is designed to preserve familiar HDFE workflow and functionality.
- Model specification: The dependent variable must be nonnegative but need not be integer-valued, while explanatory variables may use factor and time-series operators.This reflects the pseudo-Poisson setting rather than a restriction to count outcomes.
- Fixed effects: absorb() specifies categorical fixed effects, with options to save estimated fixed effects; d() creates their summed contribution for later prediction.The d() variable is required when absorbed fixed effects are followed by predict.
- Command options: Options control constrained regressors, variance estimation, debugging output, convergence tolerance, starting values, iteration limits, and separation handling.Separation checks support fe, ir, simplex, and mu methods, while none disables all checks.
- Command options: The command defaults to a 1e-8 convergence criterion, permits up to 10,000 iterations, and reports exponentiated coefficients through the irr option.Additional options govern singleton retention, version reporting, and iteration-output visibility.
- Stored results: ppmlhdfe stores coefficients, variance-covariance matrices, likelihood statistics, iteration counts, fixed-effect degrees of freedom, clustering information, and sample indicators in e().Stored results also identify separated and dropped singleton observations, model variables, absorbed effects, and prediction-related programs.
5 Examples
The examples show ppmlhdfe handling separation, reproducing standard fixed-effects Poisson results, absorbing multiple high-dimensional effects, and estimating large gravity and citation models.
- Separation and estimation: ppmlhdfe drops separated observations and collinear regressors, producing more plausible estimates where glm fails to converge and ppml yields unreliable results.In the example, it drops one observation and x2, then converges in 6 iterations and 6 HDFE sub-iterations.
- Fixed-effects equivalence: The ships example reproduces xtpoisson’s coefficient estimates while reporting robust standard errors and the actual Poisson log-likelihood.The model uses ship as a fixed effect and service as an exposure control.
- Multiple fixed effects: Absorbing three fixed-effects groups in the ships example yields the same results for explicitly retained variables as including those categorical effects directly.The specification absorbs ship, co_70_74, and co_75_79 while retaining op_75_79 and co_65_69.
- Implementation: The HDFE-IRLS algorithm combines within-transformation and weighted regression, using reghdfe’s computational machinery where possible.The authors describe the core algorithm as portable to languages offering robust within-transformation and standard weighted-regression routines.
- Large applied models: The citation example demonstrates flexible journal, article, JEL-code, and publication-year specifications, with the table reporting a robust impact of author count on citations.The specifications vary by journal effects, journal-specific trends, and journal-by-year effects.
6 Conclusion
The conclusion presents ppmlhdfe as a fast Stata command for Poisson models with multiple HDFE. It combines reghdfe-like functionality with stronger existence checks and acceleration techniques, with possible extension to other GLM models.
- Conclusion: ppmlhdfe enables fast estimation of (pseudo) Poisson regression models with multiple high-dimensional fixed effects.It has syntax and many functionalities similar to reghdfe.
- Conclusion: The command checks whether maximum-likelihood results exist and introduces techniques for accelerating nonlinear estimation with high-dimensional covariates.The conclusion characterizes these as promising new concepts.
- Conclusion: The estimation approach could be extended to other models in the generalized linear model family.The paper states this as a potential extension rather than an implemented capability.