Source-linked AI summary
NGBoost: Natural Gradient Boosting for Probabilistic Prediction
Tony Duan, Anand Avati, Daisy Yi Ding, Khanh K. Thai, Sanjay Basu, Andrew Y. Ng, Alejandro Schuler
TL;DR
Many regression applications need predictive uncertainty and flexible probability queries beyond a single point prediction. NGBoost treats distribution parameters as jointly boosted targets and uses the natural gradient to correct training dynamics, achieving competitive probabilistic and point estimates relative to other models.
Problem
Regression applications often require uncertainty quantification and probability queries, but standard methods typically return only a single best-guess prediction.
Method
NGBoost jointly estimates conditional-distribution parameters with multiparameter boosting and uses the natural gradient to pre-scale gradients through the inverse Fisher Information.
Results
NGBoost performs competitively with other models in predictive uncertainty and point estimates, while its natural-gradient and multiparameter components improve performance together.
Takeaways & Limitations
NGBoost provides a common, flexible, scalable, and easy-to-use framework for probabilistic prediction across classification, regression, and survival problems.
Takeaways & Limitations
Finite step sizes remove the natural gradient’s invariance property, and computational cost increases linearly with distributional parameters while natural-gradient computation scales with p^3 and linearly with N.
Abstract
from arXiv · showhide
We present Natural Gradient Boosting (NGBoost), an algorithm for generic probabilistic prediction via gradient boosting. Typical regression models return a point estimate, conditional on covariates, but probabilistic regression models output a full probability distribution over the outcome space, conditional on the covariates. This allows for predictive uncertainty estimation -- crucial in applications like healthcare and weather forecasting. NGBoost generalizes gradient boosting to probabilistic regression by treating the parameters of the conditional distribution as targets for a multiparameter boosting algorithm. Furthermore, we show how the Natural Gradient is required to correct the training dynamics of our multiparameter boosting approach. NGBoost can be used with any base learner, any family of distributions with continuous parameters, and any scoring rule. NGBoost matches or exceeds the performance of existing methods for probabilistic prediction while offering additional benefits in flexibility, scalability, and usability. An open-source implementation is available at github.com/stanfordmlgroup/ngboost.
1. Introduction
Probabilistic regression estimates conditional outcome distributions rather than point predictions, enabling uncertainty quantification and flexible probability queries. NGBoost addresses limitations in existing probabilistic regression methods by extending modular gradient boosting with multiparameter boosting and natural gradients.
- Weather forecasting and clinical prediction are important supervised regression applications.
- Most machine learning regression methods return a single best-guess prediction, limiting direct uncertainty quantification.
- Probabilistic regression estimates P(y|x), enabling arbitrary event-probability queries instead of only E[y|x].
- Existing probabilistic regression methods can be inflexible, slow, or inaccessible to non-experts.
- NGBoost addresses the challenge of jointly boosting multiple distribution parameters using multiparameter boosting and natural gradients.
2. Summary of Contributions
NGBoost combines multiparameter boosting and generalized natural gradients in a modular probabilistic-regression algorithm. The paper reports competitive predictive uncertainty and point estimates relative to other models.
- NGBoost is a modular algorithm for probabilistic regression using multiparameter boosting and natural gradients.
- The method integrates any choice of base learner, parametric probability distribution, and scoring rule.
- NGBoost generalizes the natural gradient to scoring rules such as CRPS.
- NGBoost performs competitively relative to other models in predictive uncertainty estimates and point estimates.
3. Natural Gradient Boosting
NGBoost extends gradient boosting to estimate conditional probability distributions by jointly boosting distribution parameters under proper scoring rules. Natural gradients correct parameterization-dependent and imbalanced updates, enabling flexible and stable probabilistic prediction with modular learners, distributions, and scores.
- Probabilistic regression estimates Pθ(y|x) by modeling the distribution’s p parameters as functions of x, rather than estimating only E[y|x].
- Proper Scoring Rules: Proper scoring rules compare forecast distributions with observed outcomes; NGBoost supports logarithmic scoring, which yields MLE, and CRPS, a robust alternative.
- The Generalized Natural Gradient: The ordinary gradient depends on parameterization, whereas the generalized natural gradient is invariant to parameterization and follows steepest ascent in Riemannian space.
- NGBoost: Natural Gradient Boosting: NGBoost jointly estimates distribution parameters with boosting, combining additive base-learner outputs with an initial parameter vector to produce conditional distributions.
- Extensions and Complexity: NGBoost accepts arbitrary base learners, continuous-parameter distribution families, and scoring rules, while its computational cost grows linearly with parameter count and cubically with matrix dimension per observation.
- Multiparameter Boosting: Natural-gradient updates pre-scale examples and parameters, keeping convergence balanced under a common stage multiplier and avoiding the ordinary gradient’s suboptimal dynamics.
4. Experiments
The experiments evaluate NGBoost on UCI regression benchmarks for probabilistic and point prediction, including comparisons with established methods and ablations of its components. NGBoost is competitive for probabilistic prediction, especially on smaller datasets, and for point estimation despite not being optimized for it.
- Experimental protocol: The evaluation uses repeated train-validation-test splits on UCI datasets, with 10% held out for testing and validation selecting the number of boosting stages.The procedure is repeated 20 times for most datasets, five times for Protein, and once for Year MSD.
- Probabilistic regression: Table 1 measures probabilistic regression with test negative log-likelihood and compares NGBoost with MC dropout, Deep Ensembles, Concrete Dropout, Gaussian Processes, GAMLSS, and Distributional Forests.The Year MSD results are unavailable for some methods because required gradient-based hyperparameter computations did not fit in memory.
- Probabilistic regression: NGBoost offers competitive negative log-likelihood performance, especially on smaller datasets.Table 1 bolds the best method and methods whose standard errors overlap with the best.
- Ablations: The ablation compares full NGBoost with second-order boosting, ordinary multiparameter boosting, and homoscedastic boosting to isolate natural-gradient and distributional-parameter effects.The reported ablation conclusion is that multiparameter boosting must be used together with the natural gradient to increase performance.
- Experimental protocol: NGBoost uses Normal distributions, depth-three decision trees, log scoring, and generally a learning rate of 0.01.Year MSD uses learning rate 0.1 and 10% mini-batches because it is much larger than the other datasets.
- Point estimation: Table 3 evaluates extracted expectation estimates using test RMSE against elastic net, random forests, and standard gradient boosting.Although NGBoost is not optimized for point estimation, the paper reports competitive point-estimation performance.
5. Conclusions
NGBoost combines multiparameter boosting with natural gradients for flexible probabilistic prediction, achieving competitive performance while retaining practical advantages. Its empirical benefits are established, but theoretical inference guarantees and some technical improvements remain open.
- Conclusions: NGBoost combines multiparameter boosting with the natural gradient to estimate outcome-distribution parameters as functions of observed features.The natural gradient corrects the training dynamics of multiparameter boosting.
- Conclusions: NGBoost performs as well as existing probabilistic-regression methods while remaining flexible, scalable, and easy to use.It supports multiple problem types through the same software package and interface and scales with traditional boosting complexity.
- Conclusions: The natural gradient and multiparameter boosting work together to improve performance, often outperforming homoscedastic modeling by a large margin.Ablations indicate that the improvement comes from exploiting curvature in distributional space rather than parameter space.
- Conclusions: NGBoost’s probabilistic predictions impose only a modest RMSE loss compared with dedicated point-prediction models.The reported point-estimation comparison is against elastic net, random forests, and gradient boosting.
- Conclusions: The framework is also applicable to classification, right-censored survival problems, and joint prediction of multiple outcomes.Joint prediction can represent questions involving events across outcomes such as rainfall and temperature.
- Conclusions: Open work includes higher-order invariance, improved tree learners and regularization, and theoretical guarantees for inference and consistency.The paper specifically leaves convergence conditions, uniform convergence, and misspecification behavior unresolved.