PaperlayerView exact revision on arXiv

Source-linked AI summary

Dirichlet Process Mixtures of Generalized Linear Models

Lauren A. Hannah, David M. Blei, Warren B. Powell

arXiv:0909.5194v2stat.ML

TL;DR

The paper addresses the limitation that GLMs model covariate effects linearly, proposing DP-GLMs that mix local GLMs for flexible Bayesian nonparametric regression. It establishes conditions for asymptotic unbiasedness and reports competitive performance across regression settings, while noting computational and dimensionality limits.

  • Problem

    GLMs impose linear covariate effects, motivating nonparametric regression methods that can represent nonlinear response functions.

  • Method

    DP-GLMs mix local generalized linear models within a Bayesian nonparametric framework to accommodate varied response and covariate types and heteroscedasticity.

  • Results

    The DP-GLM was competitive in most settings and provided stable, conservative estimates, including with extremely small sample sizes.

  • Takeaways & Limitations

    DP-GLMs provide one model and corresponding inference algorithms that perform well across many regression settings.

  • Takeaways & Limitations

    MCMC implementation does not scale well to large datasets, and the method is not appropriate for high-dimensional covariates because the covariate posterior can swamp the response posterior.

Abstract

from arXiv · show

We propose Dirichlet Process mixtures of Generalized Linear Models (DP-GLM), a new method of nonparametric regression that accommodates continuous and categorical inputs, and responses that can be modeled by a generalized linear model. We prove conditions for the asymptotic unbiasedness of the DP-GLM regression mean function estimate. We also give examples for when those conditions hold, including models for compactly supported continuous distributions and a model with continuous covariates and categorical response. We empirically analyze the properties of the DP-GLM and why it provides better results than existing Dirichlet process mixture regression models. We evaluate DP-GLM on several data sets, comparing it to modern methods of nonparametric regression like CART, Bayesian trees and Gaussian processes. Compared to existing techniques, the DP-GLM provides a single model (and corresponding inference algorithms) that performs well in many regression settings.

1 Introduction

The paper develops DP-GLMs to overcome GLMs’ linear-covariate restriction while retaining GLM response flexibility. The method mixes local GLMs using a Bayesian nonparametric model to represent nonlinear, heterogeneous regression functions.

  • GLMs restrict covariates to affect the response distribution through a linear function, limiting applications with nonlinear covariate effects.
  • The proposed method targets arbitrary response shapes, diverse response and covariate types, high-dimensional covariates, and heteroscedasticity.
  • DP-GLMs model the mean function by mixing local GLM response distributions, producing nonlinear overall means after marginalizing over local-regression uncertainty.
  • A Bayesian nonparametric mixture lets the data determine the number and form of local regressions, using more components for complex functions and fewer for simple ones.
  • The paper derives Gibbs samplers, studies posterior and asymptotic-unbiasedness properties, and evaluates DP-GLMs across several data types.
  • The unbiasedness conditions are model-dependent and can be difficult to check.

2 Related work

Prior Bayesian nonparametric regression work includes Gaussian processes, Bayesian trees, and Dirichlet process mixtures, but coverage of response types and asymptotic properties remains limited. DP-GLM extends related DP regression ideas to broader response distributions.

  • Gaussian processes, Bayesian regression trees, and Dirichlet process mixtures are common Bayesian nonparametric regression priors.
  • Joint Gaussian-mixture approaches primarily address continuous responses, while dependent-DP extensions model multiple response functionals.
  • Research on Bayesian nonparametric models for response types beyond continuous outcomes is comparatively limited.
  • Several DP-based GLM extensions retain the assumption that all covariates enter linearly in the same way.
  • Asymptotic properties of Dirichlet process regression models have received relatively little study compared with posterior-density consistency.

3 Mathematical background

The mathematical background introduces GLMs and Dirichlet process mixtures. GLMs connect linear predictors to response distributions through link functions, while DPs provide random discrete distributions that induce clustering and an unknown number of groups.

  • Dirichlet Process Mixture Models: A Dirichlet process is a distribution over distributions, parameterized by a base distribution G0 and positive concentration parameter α.
  • Dirichlet Process Mixture Models: DP draws are discrete and place mass on a countably infinite collection of atoms drawn from G0.
  • Dirichlet Process Mixture Models: The Polya-urn conditional distribution allows observations to share parameter values, creating clusters whose number is random and unknown.
  • Dirichlet Process Mixture Models: In a DP mixture, θ is a latent parameter associated with an observed data point, and posterior clustering groups observations by shared parameters.
  • Generalized Linear Models: GLMs combine a conditional response-probability model, a linear predictor, and a link function.
  • Generalized Linear Models: A GLM uses a link function to relate the conditional mean to the linear predictor Xβ, with µ = g^-1(Xβ).

4 Dirichlet process mixtures of generalized linear models

DP-GLMs combine local GLMs in a Dirichlet process mixture, allowing flexible response distributions across covariate regions. The model supports varied covariate and response types, heteroscedasticity, posterior prediction, and improved resistance to dimensionality relative to DPMM regression.

  • Model formulation: DP-GLMs model covariate distributions with exponential-family mixtures and responses with GLMs, linking each mixture component to its own GLM coefficients.This connects local covariate regions with response-specific regression relationships.
  • Model formulation: The framework accommodates continuous, categorical, count, and other response types by selecting the conditional GLM and corresponding covariate representation.Examples include Gaussian models for continuous data and multinomial models for categorical covariates or responses.
  • Heteroscedasticity and overdispersion: Mixture-specific variance parameters allow multiparameter GLMs to model heteroscedasticity through smoothly transitioning posterior response distributions.The DP-GLM is described as producing smooth mean estimates while remaining robust to heteroscedastic data.
  • Posterior prediction with a DP-GLM: The base measure controls model expressiveness, computational efficiency, and whether properties such as asymptotic unbiasedness hold.For the Gaussian model, a conjugate normal-inverse-gamma construction supports continuous integrable distributions and efficient collapsed Gibbs sampling, but can be less expressive with little data.
  • Comparison to the Dirichlet process mixture model regression: As dimensionality grows, DP-GLM predictions are more resistant than DPMM predictions because response-related parameters retain greater influence on mixture selection.Each additional coefficient can add noise, making DP-GLM estimates noisy when the covariate dimension is large.

5 Asymptotic Unbiasedness of the DP-GLM Regression Model

The paper studies whether DP-GLM mean-function estimates become asymptotically unbiased and provides sufficient conditions, proofs, and examples for that property. The analysis combines posterior weak consistency with uniform integrability and examines Gaussian and multinomial cases.

  • Motivation: Asymptotic unbiasedness is needed because Dirichlet process priors do not automatically make posterior regression estimates converge to the true mean function.The paper motivates this concern with a prior example whose posterior location estimate remained asymptotically biased.
  • General conditions: The proof strategy establishes weak posterior consistency for the joint distribution and uses uniform integrability to obtain convergence of conditional expectations.Uniform integrability also ensures that the posterior mean-function estimate exists for every sample size under the stated conditions.
  • General conditions: Under Theorem 5.1’s conditions, the posterior conditional mean exists for every n and converges almost surely to the true conditional expectation.The result is stated for covariates x in a compact set and a prior Πf on the density class F.
  • Gaussian example: For the Gaussian model, compact support of the true joint density, weak support for all location-scale-slope mixtures, and the integrability condition imply Theorem 5.1’s conclusions.The paper presents Normal-Inverse-Wishart, diagonal Normal-Inverse-Gamma, and Normal Mean, Log Normal Variance base-measure examples.
  • Multinomial example: For the multinomial model, compactly supported covariates, continuous class probabilities, weak support for location-scale-slope mixtures, and the integrability condition imply the same conclusions.The paper gives corresponding base-measure examples, using Gaussian GLM parameters and continuous-covariate constructions from the Gaussian model.

6 Empirical study

The empirical study compares DP-GLM with regression methods across datasets spanning heteroscedasticity, moderate dimensionality, diverse input types, and count responses. DP-GLM is competitive across these settings, with especially strong performance on the Solar data and smaller CCS training sets.

  • Datasets: The evaluation covers three datasets designed to test heteroscedasticity, moderate dimensionality, diverse input and response types, and small-to-moderate training sizes.Training sizes ranged from 20 to 800 observations.
  • Competitors: The competitor set includes tree methods, Gaussian processes, ordinary least squares, basic DP regression, and Poisson GLM, with applicability varying by regression problem.Gaussian processes support only continuous inputs and responses, whereas CART accommodates categorical and continuous inputs and any response type.
  • CMB results: On CMB, all nonlinear methods except CART performed comparably, while OLS performed poorly because the dataset was nonlinear.CART had difficulty finding an appropriate bandwidth, and several methods—including DP-GLM—fit heteroscedasticity.
  • CCS results: On CCS, Gaussian processes performed best overall, while DP-GLM outperformed frequentist CART on smaller training sets.Linear regression, basic DP regression, and Bayesian CART performed comparatively poorly; treed methods often had convergence problems.
  • Solar results: On Solar, DP-GLM had the best performance under both error measures among methods able to handle the categorical covariates and count response.The high mean squared error suggested frequentist CART overfit, while Poisson regression inadequately fit nonlinearities according to mean absolute error.
  • Overall comparison: Across datasets, DP-GLM was a relatively strong competitor and more stable than most Bayesian competitors on CCS, aside from Gaussian processes.The discussion identifies CART and Bayesian CART as the only methods matching DP-GLM’s flexibility.

7 Conclusions and Future Work

The paper concludes that DP-GLM is a flexible, stable Bayesian regression technique with theoretical unbiasedness results and competitive empirical performance. It also identifies computational scalability and high-dimensional covariate settings as important boundaries.

  • Conclusions: DP-GLM combines a flexible Bayesian regression model with statistical analysis, asymptotic-unbiasedness conditions, and evaluation against state-of-the-art Bayesian competitors.The paper gives situations in which the unbiasedness conditions hold.
  • Conclusions: DP-GLM was competitive in most settings and provided stable, conservative estimates even with extremely small sample sizes.
  • Future work: The implementation relies on MCMC, which does not scale well to large datasets, motivating variational inference as a possible alternative.
  • Future work: Joint posterior modeling can allow the covariate posterior to swamp the response posterior, although careful modeling can mitigate this at low to moderate dimensionality.

Posterior Inference

Posterior inference uses Gibbs sampling over cluster labels and component parameters. The sampler incorporates covariate and response likelihood terms, with some parameters integrated out when conjugacy permits.

  • Sampler: The Gibbs sampler state consists of cluster labels (z1, ..., zn) and component parameters.
  • Algorithm: Algorithm 1 iterates label and parameter updates until convergence criteria are met and then obtains M posterior samples.
  • Sampler: The DP-GLM sampler updates each observation’s label using a Chinese Restaurant Process term and covariate and response likelihood terms.The response likelihood term is novel relative to traditional Dirichlet process mixture samplers.
  • Collapsed inference: In collapsed Gibbs sampling, all or part of the component parameters are integrated out when the model and base measure allow it.The text gives conjugate Gaussian models with Normal-Inverse-Gamma base measures as an example.

A-2 Proof of Theorem 5.1

The proof establishes asymptotic convergence of the DP-GLM regression mean by combining posterior consistency, conditional-density convergence, and uniform integrability. Its steps rely on conditions ensuring consistency, existence of the limiting mean, and finite mean estimates.

  • Proof strategy: The proof proceeds in two stages: show consistency, then use conditions (ii) and (iii) to obtain existence and convergence of the expectation.The second stage uses weak convergence plus uniform integrability to pass from conditional densities to conditional means.
  • Proof strategy: The proof first establishes posterior consistency under condition (i), using Schwartz’s theorem when the prior places positive probability on suitable neighborhoods of f0.Condition (i) is the difficult condition to verify, but related Dirichlet process mixture priors have established weak-consistency results.
  • Proof strategy: Conditions (ii) and (iii), together with conditional-density convergence, ensure existence and convergence of the posterior conditional expectations.Condition (ii) guarantees a limiting mean function, while condition (iii) supports finite mean estimates and uniform integrability.
  • Proof strategy: Weak convergence of the joint density and its marginal implies weak convergence of the conditional density fn(y|x) to f0(y|x).The conditional-density result is formalized in Lemma A-2.

A-3 Proof of Theorem 5.2 and Results for Base Measures

Theorem 5.2 is proved by approximating compactly supported continuous densities with Gaussian convolutions and verifying the required prior-support conditions. The proposed base measures satisfy the theorem because they place support over the desired parameter space.

  • Proof of Theorem 5.2: The proof uses Gaussian convolution approximations to f0 and establishes posterior weak consistency for the approximating densities.The argument is sketched first for scalar x and extends to vector-valued covariates.
  • Proof of Theorem 5.2: Uniform equicontinuity and the Arzela-Ascoli theorem provide a finite tiling that approximates the target function arbitrarily well on a bounded region.The remaining parts of the proof follow the approximation strategy of Ghosal et al. (1999).
  • Proof of Theorem 5.2: Compact support allows the proof to partition the relevant (x,y) space and bound the convolution approximation over central and tail regions.The construction controls the approximation using bounded parameter ranges and Gaussian variances in the prior’s weak support.
  • Results for base measures: Theorem A-5 extends to scale mixtures of σx and σy, and Theorem 5.2 follows directly from Theorem A-5.The approximation result therefore covers the Gaussian model under the stated compact-support and weak-support assumptions.
  • Results for base measures: All base measures proposed in Subsection 5.2 have weak support over the desired parameter space and hence satisfy Theorem 5.2.The supported space is Rd × Rd+1 × Rd+1+ as represented in the passage.

A-4 Proof of Theorem 5.3 and Results for Base Measures

Theorem 5.3 is established for continuous covariates with categorical responses by approximating conditional probabilities and then approximating them with GLM-form densities. The specified base measures have sufficient weak support, while experiments describe sampling settings and model behavior.

  • Proof of Theorem 5.3: The proof first constructs compact-support approximations for the joint density f0(x,k) and controls the corresponding log-density approximation.This follows the same general plan as the Gaussian proof.
  • Proof of Theorem 5.3: Conditional probabilities are truncated away from 0 and 1 before being approximated by densities with the desired GLM form.For binary outcomes, probabilities below γ are raised to γ and probabilities above 1−γ are lowered to 1−γ.
  • Proof of Theorem 5.3: Continuity on a compact covariate domain supplies a finite Lipschitz bound, while bounded GLM parameters and equicontinuity complete the approximation argument.The construction chooses finite bounds b0 and b1 and uses equicontinuity over the resulting parameter set.
  • Results for base measures: Theorem 5.3 follows from Theorem A-7, and the base measures supplied for the model have sufficient weak support.This parallels the Gaussian example’s support argument.
  • Empirical settings: The reported implementations used burn-in and posterior sampling schedules, with runtime varying by dataset and implementation.One Matlab implementation took about 500 seconds on the largest dataset, while another reimplementation reported longest runtimes under about 10 seconds.
  • Empirical settings: The conjugate base-measure model failed to capture heteroscedasticity well on one dataset, so subsequent tests there used the log-normal base measure.The passage reports a dataset-specific modeling choice rather than a general theoretical limitation.
Loading 0909.5194v2…