PaperlayerView exact revision on arXiv

Source-linked AI summary

LiD-GLM: Lipschitz-constrained Deep Generalized Linear Models

Tom Splittgerber, Niklas Koenen, Marvin N. Wright, Werner Brannath

arXiv:2608.16340v1stat.MLcs.LG

TL;DR

Expressive models often require structural constraints that limit flexibility, while weaker constraints leave inherent interpretability uncertain. LiD-GLM uses Lipschitz-constrained invertible residual networks to extend GLMs with bounded nonlinear and distributional corrections, and the authors report that it can bridge restrictive GLMs and unconstrained machine-learning methods on moderately nonlinear, non-additive data.

  • Problem

    Existing expressive models often require strict structural constraints that limit flexibility, while weaker constraints leave inherent interpretability uncertain.

  • Method

    LiD-GLM extends generalized linear models with Lipschitz-constrained invertible residual networks and post-hoc orthogonalization for bounded flexibility and identifiability.

  • Results

    On moderately nonlinear and non-additive datasets, LiD-GLM closed the gap between restrictive GLMs and over-expressive unconstrained machine-learning methods.

  • Takeaways & Limitations

    Researchers can vary the Lipschitz constant to choose an informed degree of nonlinearity when assessing whether GLM data benefit from moderate nonlinearization.

  • Takeaways & Limitations

    The model’s approximation capability depends on input scaling and activation choice, and some constrained architectures cannot approximate all functions with Lipschitz constant one.

Abstract

from arXiv · show

The combination of traditional statistical models and neural network (NN) components into semi-structured hybrid models is an intriguing approach to construct models that, ideally, combine traditional interpretability with the unprecedented flexibility of NNs. In order to preserve interpretability, it is usually necessary to restrict the NN components to prevent them from dominating the model. However, existing methods that enforce structural constraints on their NN components severely limit their models' flexibility; in contrast, methods that only enforce weak, indirect constraints lose meaningful interpretability. The method we propose therefore leverages invertible residual neural networks (i-ResNets) to equip generalized linear models with both nonlinear parameter estimation and a flexible correction of their distributional assumptions while always retaining stochastic monotonicity of the modeled distribution in the (formerly linear) predictor. The i-ResNets correspond to a controlled deviation from identity and by constraining their Lipschitz constant one can rigorously limit and quantify how far the hybrid model deviates from its traditional counterpart. This enables a user-specifiable compromise between flexibility and interpretability without limiting the structure of nonlinear and interaction effects that can be learned. Furthermore, we develop specific inherent interpretation techniques for our model and enforce model identifiability through an adapted post-hoc orthogonalization.

1 Introduction

LiD-GLM extends generalized linear models with controllable neural flexibility while preserving inherent interpretability and stochastic monotonicity. It models full conditional distributions, relaxes distributional assumptions through latent-variable flows, and addresses identifiability with post-hoc orthogonalization.

  • Motivation and contribution: The paper proposes a conditional distribution model with an explicit, controllable trade-off between expressiveness and interpretability.Modeling the full conditional distribution also enables precise description of aleatoric uncertainty.
  • Method overview: LiD-GLM integrates one i-ResNet for nonlinear predictor extension and a second i-ResNet for relaxing distributional assumptions through a latent-variable representation.The latent-variable construction supports normalizing-flow transformations, including for discrete exponential families where direct monotone transformations would alter support.
  • Method overview: Lipschitz constraints limit how far the neural extensions deviate from the GLM and its distributional assumptions without requiring a predefined structure for nonlinear and interaction effects.Each residual block has the form Id + ν, with L(ν) ≤ c and c < 1.
  • Interpretability: The model preserves stochastic monotonicity of Y in the predictor η because the distributional i-ResNet is invertible and the latent transformation is increasing in θ.The predictor transformation also remains monotone in each transformed input term, although η need not generally be monotone in each covariate.
  • Interpretability: Identifiability is enforced through an adapted PHO orthogonalization, complemented by interpretational methods evaluated on real and synthetic applications.The paper also discusses comparisons with comparable models across datasets of different complexity.

2 Lipschitz-constrained DeepGLM (LiD-GLM)

LiD-GLM extends generalized linear models with i-ResNets for nonlinear and interaction effects, while user-controlled Lipschitz bounds regulate deviation from the linear predictor. A second i-ResNet corrects distributional assumptions, and post-hoc orthogonalization enforces identifiability.

  • Nonlinear prediction: An input i-ResNet transforms covariates before the GLM component, enabling nonlinear and interaction effects in the prediction of η.The transformed covariates are z = T_p(x).
  • Lipschitz constraint: The user-defined bound L_b^p limits the impact of the nonlinear residual and thereby controls the model’s nonlinearity and deviation from a constant.The bound regulates how much the residual changes with covariates, rather than directly limiting its absolute size.
  • Lipschitz constraint: When L_b^p < 1, each transformed coordinate remains monotonically increasing in its corresponding input coordinate.This follows because the partial derivative of T_p,i(x) with respect to x_i is guaranteed to be positive.
  • Distributional correction: A second i-ResNet T_d applies a θ-independent normalizing-flow correction to the latent variable distribution, extending the GLM’s distributional assumptions.For L_b^d = 0, the model retains the base normal target distribution; increasing L_b^d permits stronger corrections.
  • Identifiability: Post-hoc orthogonalization projects ν_p onto the training-data column space, adjusts β for the corresponding linear component, and subtracts that projection from ν_p.This procedure is introduced to achieve full identifiability after nonlinear and linear effects could otherwise overlap.

3 Interpretation Methods

The section presents model-specific interpretation methods based on orthogonalization, transformed covariates, variance decomposition, and the monotone latent transformation Td. These methods quantify how closely the model retains traditional GLM interpretations while revealing nonlinear distributional effects.

  • Interpretation prerequisites: Interpretation begins by applying the Section 2.3 orthogonalization method before introducing model-specific methods beyond generic post-hoc tools.The section retains simplified coefficient notation for readability.
  • Covariate effects: In model M1, βi describes the effect of transformed covariate zi = xi + νp,i(x), while the relationship between xi and η may be complex and nonmonotone.The shared coefficient places the linear and nonlinear terms on a common scale.
  • Covariate effects: When L(νp) < 1, each transformed covariate zi is strictly monotonically increasing in xi; the same guarantee follows when the corresponding component’s Lipschitz norm is below 1.The tightness of the bound depends on the selected vector norms.
  • Variance-based interpretation: An R2-style variance decomposition quantifies how much zi is dominated by xi and therefore how well βi continues to describe xi’s effect on Y.After post-hoc orthogonalization, xi is the ordinary-least-squares prediction from the regression zi ∼ xi, linking R2_i to its traditional definition.
  • Latent transformation Td: The invertible one-dimensional transformation Td is strictly increasing, and plotting it against the identity line provides insight into the fitted LiD-GLM.Because hθ is increasing in both arguments, monotonicity is preserved in the latent variable, making Td relatively straightforward to interpret.

4 Practical Applications

Across realistic binary and continuous datasets, LiD-GLM improved on linear baselines while retaining interpretable structure through Lipschitz control, coefficient analysis, and interaction diagnostics. The applications show that users can trade expressiveness for interpretability, while post-hoc orthogonalization improves parameter estimation.

  • Experimental setup: Nested cross-validation across relatively linear, moderately nonlinear, and clearly nonlinear datasets evaluated LiD-GLM with and without distributional correction against GLM and SDDR baselines.The datasets covered both binary and continuous outcomes.
  • Cars application: On Cars, nonlinear LiD-GLM outperformed GLM, distributional correction slightly improved performance, and larger Lipschitz bounds produced diminishing test-NLL returns.Moderate nonlinearity yielded measurable benefit, while interpretability-preserving bounds approached unconstrained-model performance.
  • Cars application: Cars models learned nonlinear trends and horsepower–cylinders interactions unavailable to additive GLMs, while stronger neural-network interactions appeared potentially suspicious.LiD-GLM coefficients remained relatively similar to the initial linear model, and R2_i values near one supported GLM-like interpretation of transformed covariates.
  • Cars application: The Cars model used only 0.99 of the permitted Lipschitz constant, indicating that most of its behavior remained linear and that larger bounds offered little additional benefit.The model therefore achieved an adequate fit without exhausting its allowed nonlinearity.
  • Stroke application: In the more nonlinear application, BMI exhibited nonlinear influence through interactions with formerly binary covariates, enabling LiD-GLM to identify possible interaction effects for downstream variable selection.Age and average glucose level remained relatively unchanged, whereas interactions likely explained smaller R2_i values for other covariates.
  • Simulations and identifiability: Simulations showed negligible coefficient-reconstruction error for the linear model and recommended LiD-GLM workflow, while PHO drastically reduced Cars coefficient-estimation variance and generally reduced absolute coefficient values.Without orthogonalization, randomly initialized models incurred significant error, especially for small p and n.

5 Discussion

LiD-GLM combines flexible nonlinear parameter estimation and distributional correction while rigorously bounding deviation from GLMs through a user-chosen Lipschitz constraint. The discussion highlights its empirical promise, practical use, input-scale and activation dependencies, and directions for uncertainty quantification.

  • Contributions: LiD-GLM uses i-ResNets to enable nonlinear parameter estimation and correction of the assumed distribution family while bounding deviation from GLMs with a user-chosen Lipschitz constraint.This preserves interpretation of the linear regression coefficients up to a degree.
  • Empirical findings: On moderately nonlinear and non-additive data, LiD-GLM can close the gap between restrictive GLMs and over-expressive unconstrained machine-learning methods.The model was also tested internally on a highly linear clinical dataset, whose results could not be published for data-protection reasons.
  • Empirical findings: Initialized with a pre-trained GLM, LiD-GLM stayed at the same weight on a highly linear clinical dataset, enabling direct tests of whether moderate nonlinearization improves fit.A negative result can provide additional justification for retaining a GLM, unlike poor performance from random initialization.
  • Limitations: Because the Lipschitz constant depends on input scale, normalization is especially important, while approximation capabilities also depend on the activation function.The discussion cites ReLU-NNs with layerwise constrained Lipschitz constants as relevant to this limitation.
  • Future work: Confidence intervals for LiD-GLM regression coefficients would be valuable, with prospective methods likely combining the Lipschitz constraint and bootstrapping.This direction follows from the preserved interpretability of the regression coefficients.
  • Practical use: Researchers can test varied Lipschitz constants and choose a model or permitted nonlinearity according to their interpretability criteria.The method is positioned as a tool for GLM users who suspect their data may benefit from slightly nonlinear models.

Supplementary Material · A Decomposition of i-ResNet · B Relation between LiD-GLM and SDDR

The supplementary material proves that any i-ResNet admits a residual-form decomposition with finite Lipschitz correction and relates LiD-GLM to semi-structured deep distributional regression. LiD-GLM appears as a restricted SDDR case for a one-parameter exponential-family distribution with a Lipschitz-constrained neural network.

  • A Decomposition of i-ResNet: Any i-ResNet can be represented as an identity map plus a neural-network correction.The proof proceeds by induction over residual blocks T_j = Id + g_j with L(g_j) < 1.
  • A Decomposition of i-ResNet: For one residual block, the decomposition is immediate because T = T_1 already has the required form.This establishes the induction base case.
  • A Decomposition of i-ResNet: Composing an additional residual block preserves the decomposition by rewriting T as (Id + g_m) ◦ (Id + ν).Here ν is the correction for the preceding m−1 blocks and has finite Lipschitz constant.
  • A Decomposition of i-ResNet: The composed network remains an identity map plus correction, with ˜ν = ν + g_m(Id + ν).The construction completes the inductive step for an i-ResNet with m blocks.
  • A Decomposition of i-ResNet: The correction has finite Lipschitz constant, satisfying L(˜ν) ≤ L(ν) + L(g_j)L(Id + ν) < ∞.Thus the decomposition holds for every i-ResNet with finitely many blocks.
  • B Relation between LiD-GLM and SDDR: A general SDDR model maps each distributional parameter θ_k through a one-to-one function h_k of a predictor η_k.The predictors may combine linear terms, structured nonlinear basis-function expansions, and neural networks.
  • B Relation between LiD-GLM and SDDR: LiD-GLM is a special SDDR case with K = 1, J = 0, L = 1, x = u, and d_1 a Lipschitz-constrained neural network.Its distribution is a one-parameter exponential family parameterized by the canonical parameter θ = θ_1.
  • B Relation between LiD-GLM and SDDR: Without the Section 2.2 extension, the SDDR function h_1 corresponds to the GLM function Λ.This identifies the baseline link-function relationship between the two model classes.

C Monotonicity · D Relation to Conditional Normalizing Flows · E Choice of Vector Norms

The model’s coordinate-wise transformation is strictly increasing when the residual network has Lipschitz constant below one, while monotonicity of the full predictor is not generally guaranteed. Equation (M2) relates to conditional normalizing flows under continuous invertibility conditions, and norm choice involves both performance and computational trade-offs.

  • C Monotonicity: When L(νp) < 1, holding all other covariates fixed makes each transformed coordinate πi ◦ Tp ◦ ιx−i strictly increasing.The proof writes the coordinate map as identity plus a perturbation whose Lipschitz constant is below one.
  • C Monotonicity: The monotonicity proof does not readily extend to L(νp) ≥ 1, although the paper expects L(νp) < 1 to be more practically relevant.The stated condition is therefore both the basis of the proof and the practically emphasized regime.
  • C Monotonicity: Monotonicity of individual transformed covariates does not generally imply monotonicity of η, because other transformed covariates and mixed-sign coefficients can intervene.The passage notes that transformed covariates z_j for j ≠ i need not be monotone in x_i, and β may contain both positive and negative components.
  • D Relation to Conditional Normalizing Flows: Equation (M2) can be interpreted as a special case of a conditional normalizing flow within the framework of Winkler et al. [2023] and Algren et al. [2023].A conditional normalizing flow learns a mapping that transforms a conditional target distribution into a known conditional distribution.
  • D Relation to Conditional Normalizing Flows: Unlike a traditional conditional normalizing flow, the model permits discrete targets and does not necessarily require hθ to be a diffeomorphism in y.For continuous y satisfying the relevant invertibility and differentiability conditions, the model can be converted to the conditional-flow form, using the reverse direction.
  • D Relation to Conditional Normalizing Flows: For discrete targets, the learned probability mass at yi is evaluated as f(yi) = F(yi) −F(yi−1).This applies to ordered support Y = {yi, i ∈ I}.
  • D Relation to Conditional Normalizing Flows: For continuous targets, the density transformation formula remains usable at relevant points because the inverse mapping is uniquely defined and differentiable in S◦.The construction relies on the assumptions imposed on hθ in Theorem 1.1.
  • E Choice of Vector Norms: Although all finite-dimensional vector-space norms are theoretically equivalent, applications favored ||.||2 over ||.||1, while general ||.||p→p-norms are NP-hard to compute and require approximation.The ||.||1→1 and ||.||∞→∞ matrix norms can instead be computed as maximum absolute row or column sums.

F Proof of Theorem 1.1

The proof constructs a quantile-based transformation from a standard normal variable to any distribution in the family, preserving monotonicity in both the parameter and latent variable. It also establishes explicit, differentiable density transformations and diffeomorphic behavior for continuous responses, supporting gradient-based computation.

  • Theorem 1.1 construction: Theorem 1.1 constructs h(θ,v) = Qθ(Φ(v)) with V standard normal so that hθ(V) has distribution fθ and h is increasing in both arguments.The construction uses G0 = Φ, the standard normal cdf, and the family’s quantile functions.
  • Differentiability of densities: The density of the discrete-response model is expressed through differences of transformed probabilities and is differentiable in both the transformation weights ω and parameter θ.The proof applies the chain rule, using differentiability of the inverse transformation and of Fθ(k) in θ.
  • Continuous-response case: For a continuous response supported on an interval (j1, j2), hθ is a diffeomorphism from R to (j1, j2).This follows because Φ maps R diffeomorphically to (0, 1), while Qθ is the differentiable inverse of Fθ on the support.
  • Continuous-response case: The continuous construction is compatible with the density transformation theorem and gradient descent because hθ and its inverse are partially differentiable in both arguments.The argument uses differentiability with respect to θ and invertibility of the Jacobian through the density fθ.

G Details on PHO

This section defines identifiability for the hybrid model with respect to its loss function and explains that PHO enforces this property through a single post-training orthogonalization step. PHO preserves both predicted means and distributions while improving interpretability of the model terms.

  • Notation: The notation distinguishes the design matrix X from the neural network output matrix νW_p(X) on the training data.X ∈ R^n×k contains the training data, while νW_p(X) contains the NN outputs.
  • Identifiability: Identifiability is defined relative to the model parameters and loss function, excluding distinct parameter configurations with equal loss that satisfy the specified conditions.The parameters are written as ((β0, β), W) and ((δ0, δ), V).
  • Post-hoc orthogonalization: PHO guarantees identifiability without changing model predictions, applying one orthogonalization step after training to improve interpretability of the model terms.Both the predicted mean and the predicted distribution remain unchanged.

G.1 Computation of PHO … H.3 Stroke Application

The paper establishes PHO-based identifiability through orthogonalization of nonlinear corrections and details the validation and training procedures used in the application studies. Cars and stroke models use task-specific distributional settings, early stopping, and validation-based model selection.

  • G.1 Computation of PHO: PHO extracts linear effects from νp using multivariate regression against the original design matrix augmented with an intercept.The fitted linear component is removed from νp to form the nonlinear correction.
  • G.1 Computation of PHO: The resulting nonlinear correction is orthogonal to the span of the original design matrix, and rescaling preserves this orthogonality.The correction is represented through the orthogonal projection P⊥_X and can be scaled to recover the model form.
  • G.1 Computation of PHO: The orthogonalized model is identifiable.Lemma G.2 follows because the relevant terms lie simultaneously in a linear span and its orthogonal complement only when both vanish.
  • G.1 Computation of PHO: PHO emphasizes the multidimensional outputs of νp because they can represent dimension-wise nonlinear correction terms with meaningful interpretations.The actual weights of eβ remain equal to those computed by R¨ugamer [2023].
  • H.1 Nested Cross Validation Details: Performance comparisons use nested cross-validation with a 5-fold outer split and a holdout-based inner loop for hyperparameter optimization.Except for the GLM, models are tuned in the inner loop and evaluated on the outer-loop test data; early-stopping models are not refit before testing.
  • H.2 Cars Application: Cars experiments compare LiD-GLM variants with and without distributional correction, using Gaussian identity-link models trained for a maximum of 3000 epochs.Training uses early stopping with a patience of 500 epochs, negative log-likelihood, and reloading of the validation-best parameters.

I Additional Results · I.1 Cars Application

This section presents additional application results not included in the main paper, focusing on LiD-GLM with distributional correction on the Cars data. The reported performance depends on L_b and follows largely the same trend as LiD-GLM without T_d.

  • I Additional Results: The section provides additional results for applications discussed in Section 4 but omitted from the main paper.
  • I.1 Cars Application: These results specifically concern the Cars application.
  • I.1 Cars Application: Figure 15 reports LiD-GLM performance with a distributional correction T_d.
  • I.1 Cars Application: The performance is evaluated using nested cross-validation on the Cars data.
  • I.1 Cars Application: Figure 15 examines performance as a function of L_b.
  • I.1 Cars Application: The performance trend is largely the same as for LiD-GLM without T_d.

I.2 Stroke Application

In the stroke application, allowing linear coefficients β to train with the neural networks improved NLL but changed the coefficients substantially. PDPs remained similar for several covariates while differing significantly for others, and R2_i values must be interpreted relative to the changed coefficients.

  • I.2 Stroke Application: 0.116 NLL was achieved when the linear coefficients β were not frozen during neural-network training.The model used the same hyperparameters as Section 4.2.2.
  • I.2 Stroke Application: The unfrozen-coefficient model changed its linear coefficients much more than the corresponding “Cars” application.Its R2_i values therefore certify similarity only to these newly changed linear coefficients.
  • I.2 Stroke Application: The R2_i value for “heart disease” was drastically larger than in the model with frozen β.This comparison concerns the stroke model’s coefficient table.
  • I.2 Stroke Application: PDPs for numeric covariates and for “hypertension”, “heart disease” and “ever married” were almost identical to prior plots modulo scale, whereas other PDPs differed significantly.The numeric-covariate PDPs appear in Figure 16, and the former binary-covariate transformations appear in Figure 17.

I.3 PHO

Post-hoc orthogonalization (PHO) nearly equally reduces variance in LiD-GLM’s estimated linear coefficients on cars with and without distributional correction. In the less suitable Stroke setting, PHO improves parameter stability selectively across covariates.

  • Cars: On the cars dataset, PHO produces nearly the same reduction in estimated linear-coefficient variance with distributional correction Td and without it.The comparison covers all data splits in both settings.
  • Stroke: On Stroke, PHO reduces parameter-estimate variance and absolute values for sex, age, hypertension, and heart disease, but not the other five covariates.Stroke is less relevant for LiD-GLMs because strong performance requires a relatively large Lipschitz constant and substantially reduces interpretability.
Loading 2608.16340v1…