Source-linked AI summary
Inference of High-dimensional Autoregressive Generalized Linear Models
Eric C. Hall, Garvesh Raskutti, Rebecca Willett
TL;DR
High-dimensional non-Gaussian autoregressive models are used to infer network dependencies, but existing theory offers limited guarantees for such dependent observations. The paper develops a sparsity-regularized likelihood estimator and derives error and sample-complexity bounds, including for Bernoulli and Poisson models. The bounds characterize how estimator error depends on sparsity, network degree, dimension, and observation time.
Problem
Existing statistical guarantees provide limited guidance for high-dimensional non-Gaussian autoregressive inference, where the data-dependent design matrix complicates LASSO-style analysis.
Source: §1 · pp. 1–3Method
The paper analyzes a sparsity-regularized maximum-likelihood estimator for vector GLAR models using martingale concentration and empirical-process techniques for dependent data.
Source: §2 · p. 4, §7.4 · p. 29Results
The paper proves estimator-error bounds as functions of sparsity, maximum node degree, dimension, and time, and derives sample-complexity results for Bernoulli and Poisson autoregressive models.
Source: §6 · p. 21Takeaways & Limitations
With fixed maximum degree, error scales inversely with observation time, linearly with sparsity, and logarithmically with dimension.
Source: §5 · p. 19The guarantees require sparsity not only in the overall network but also within each row of the autoregressive matrix.
Source: §3 · p. 5, §5.1 · p. 20
Abstract
from arXiv · showhide
Vector autoregressive models characterize a variety of time series in which linear combinations of current and past observations can be used to accurately predict future observations. For instance, each element of an observation vector could correspond to a different node in a network, and the parameters of an autoregressive model would correspond to the impact of the network structure on the time series evolution. Often these models are used successfully in practice to learn the structure of social, epidemiological, financial, or biological neural networks. However, little is known about statistical guarantees on estimates of such models in non-Gaussian settings. This paper addresses the inference of the autoregressive parameters and associated network structure within a generalized linear model framework that includes Poisson and Bernoulli autoregressive processes. At the heart of this analysis is a sparsity-regularized maximum likelihood estimator. While sparsity-regularization is well-studied in the statistics and machine learning communities, those analysis methods cannot be applied to autoregressive generalized linear models because of the correlations and potential heteroscedasticity inherent in the observations. Sample complexity bounds are derived using a combination of martingale concentration inequalities and modern empirical process techniques for dependent random variables. These bounds, which are supported by several simulation studies, characterize the impact of various network parameters on estimator performance.
1 Autoregressive Processes in High Dimensions
High-dimensional non-Gaussian autoregressive models arise in network and other time-series settings, but existing theory offers little guidance on the data needed for reliable inference. The paper develops sparse-estimation guarantees for these dependent processes, addressing challenges that standard LASSO analyses do not cover.
1 Autoregressive Processes in High Dimensions
- The analysis targets multivariate, high-dimensional autoregressive processes because Gaussian models may not capture count and other non-Gaussian observations.Examples include neural spike counts and infection cascades; the dimension can represent the number of neurons, people, or financial instruments.Source: §1 · pp. 1–2
- Existing theory provides little guidance on inference accuracy as network size, observation time, and dependency density vary.The design matrix depends on observed data and the unknown network, unlike the independent-design setting used in classical LASSO guarantees.Source: §1 · pp. 1–3
- The paper formulates a sparsity-regularized maximum-likelihood estimator and derives mean-squared-error bounds for general GLAR models.The bounds depend on problem dimension, sparsity, and the number of observations over time.Source: §1 · p. 3
- The general result yields sample-complexity bounds for Bernoulli and Poisson autoregressive models.These include the Bernoulli model related to infection cascades and the log-linear Poisson model studied alongside Hawkes and INGARCH models.Source: §1 · pp. 2–3
- The analysis combines martingale concentration, empirical-process techniques, and covering arguments to handle dependence and signal-dependent noise.Unlike the Gaussian case, the analysis cannot use linear-system spectral properties, and it must address the absence of global strong convexity.Source: §1 · p. 3
2 Problem Formulation
The vector GLAR model represents each observation component through an exponential-family conditional distribution whose natural parameter depends linearly on the previous observation vector. The paper estimates its sparse autoregressive parameter matrix with a regularized likelihood approach, under structural assumptions intended to yield small estimation error.
2 Problem Formulation
- Each component at time t+1 has an exponential-family distribution conditioned on the previous vector, with natural parameter given by an offset plus a linear function of that vector.The model assumes conditional independence among components at each time; the distribution family includes Poisson and Bernoulli cases.Source: §2 · p. 3
- The target is the autoregressive matrix A*, whose sparsity and maximum number of nonzero entries per row are structural assumptions for high-dimensional estimation.The paper denotes overall sparsity by s and the maximum row count by ρ.Source: §2 · p. 4
- The RMLE combines negative log-likelihood with an element-wise ℓ1 penalty to fit the data while favoring zero-valued entries.The estimator constrains matrix entries to specified bounds, which the paper notes can enforce stability; loose bounds facilitate analysis.Source: §2 · p. 4
- Although the model treats the offset ν as known, it can be estimated by appending it as a matrix column and appending a constant one to each observation.The paper assumes a known offset for clarity of exposition.Source: §2 · p. 4
- A least-squares LASSO formulation is inadequate here because it misses non-Gaussian observation statistics and uses a data-dependent design matrix.Classical design-matrix conditions are harder to establish when the design depends on observations and the underlying network.Source: §2 · p. 5
3 Main Results
The section derives RMLE error guarantees for high-dimensional GLAR processes under boundedness, strong-convexity, and conditional covariance assumptions, then instantiates them for Bernoulli and Poisson models. Simulations show the predicted error scaling and similar behavior with and without mixing.
3 Main Results
- Under Assumption 1, the RMLE admits row-wise error guarantees when most observations are bounded, the log-partition function is strongly convex on a bounded domain, and conditional design covariance has a positive eigenvalue lower bound.The Bernoulli and Poisson cases satisfy the assumptions with high probability.Source: §3 · pp. 5–6
- Theorem 1’s rates match in their dependence on sparsity, sample size, and regularization, but differ in their dependencies on ξ, σ, and ω.The authors attribute the differences mainly to the broader distribution family and leave rate optimality in the general setting open.Source: §3 · p. 6
3.1 Example 1: Bernoulli Distribution
- For Bernoulli GLAR, naturally bounded observations give ξ=1, and the paper derives RMLE error bounds in a corollary.The Bernoulli model also has an explicitly derived strong-convexity parameter and conditional covariance eigenvalue lower bound.Source: §3.1 · pp. 6–7
3.2 Example 2: Poisson Distribution
- The Poisson corollary gives an RMLE error rate for models whose true matrix has only non-positive entries.The analysis bounds observations by C log(MT) with high probability and by U on at least αMT indices.Source: §3.2 · pp. 7–8
3.3 Experimental Results
- MSE decays as 1/T when λ is proportional to T^-1 and grows approximately linearly with sparsity s, matching the theory.The plots report medians over 100 trials with middle-50-percentile error bars.Source: Figure 1 · p. 8, §3.3 · p. 9
- With more observations, estimated matrices approach the ground truth, while much of the remaining error comes from off-support elements.Figure 2 compares the ground truth with estimates from increasing amounts of data.Source: §3.3 · p. 9, Figure 2 · p. 9
- Allowing the process to mix before estimation produces error scaling and magnitude approximately like the experiment without mixing.The authors also state that their results require no stationarity or mixing-time assumptions.Source: §3.3 · p. 9, Figure 3 · p. 10
4 Proofs
The proofs build estimator error bounds by decoupling the likelihood by rows and using concentration arguments for dependent observations. They also establish covariance lower bounds for Bernoulli and Poisson autoregressive processes.
4.1 Proof of Theorem 1
- Theorem 1’s proof bounds row-wise error and combines the bounds into an overall error bound with high probability.The argument uses strong convexity to relate Bregman divergence to squared error, alongside martingale concentration and empirical-process techniques.Source: §4.1 · pp. 10, 12, 13, 15
4.2.1 Part 1
- For Bernoulli autoregression, the conditional variance yields a lower bound on the smallest eigenvalue of Γt.The bound is (3 + exp(ν̃ + ρ̃a))^-1.Source: §4.2.1 · p. 15
4.2.2 Part 2
- The Bernoulli Part 2 proof uses bounded martingale increments, Markov’s inequality, and a union bound over indices to derive its probability bound.It also invokes Lemma 5 and optimizes η to tighten the bound.Source: §4.2.2 · pp. 15–17
4.3.1 Part 1
- For the Poisson process, the covariance lower-bound argument uses consecutive observations in the bounded set to control the minimum eigenvalue.The proof considers observations bounded by U and uses the condition that at least ξT observations lie in the bounded set.Source: §4.3.1 · p. 17
4.3.2 Part 2
- The Poisson Part 2 proof combines bounded martingale increments, Markov’s inequality, and a union bound to derive its probability bound.The argument assumes bounded counts and uses Lemma 5, then optimizes η.Source: §4.3.2 · pp. 17–19
5 Discussion
The discussion identifies sparsity and boundedness conditions that shape estimation guarantees, distinguishes the first-order theory from open higher-order extensions, and explains how RMLE guarantees relate to stationarity.
5 Discussion
- With fixed maximum degree ρ, error scales inversely with T, linearly with sparsity s, and logarithmically with dimension M, despite estimating M² parameters.The discussion says T then needs to be on the order of log(M); the regularization parameter λ should be small for lower error but at least about T^-1/2 for the bounds to hold.Source: §5 · p. 19
5.1 Dense rows of A∗
- The theory requires both overall sparsity and sparse rows: a large star network can make the hub's inhibition and outgoing influence difficult to estimate.For that example, ρ = M and s = 2M −1; the discussion notes that power-law degree networks may keep ρ growing slowly with M.Source: §5.1 · pp. 19–20
5.2 Bounded observations and higher-order autoregressive processes
- Assumption 1's bounded-observation condition supports the concentration analysis; it is natural for Bernoulli data but no fixed observation bound holds with high probability for Poisson data.The proof restricts its error characterization to the set T defined by Assumption 1 to avoid a bound that grows polynomially with T.Source: §5.2 · p. 20
- Higher-order GLAR extensions remain unresolved because identifying the analogue of T and proving the key lemmas and Assumption 1 after vector-stacking are open questions.The discussion says much of the proof technique may still apply, especially to Bernoulli autoregression, where T is easily defined.Source: §5.2 · p. 20
5.3 Stationarity
- Under symmetric A*, the process is reversible with a stationary distribution; for countable state spaces, mixing-time bounds are available, but RMLE need not wait for stationarity.The discussion says its sample requirement scales roughly as ρ³ log(M), and RMLE guarantees also apply in settings without the stated mixing-time guarantees.Source: §5.3 · pp. 20–21
6 Conclusions
The paper establishes non-asymptotic error bounds for sparse GLAR estimation, with bounded-degree networks requiring data on the order of ρ^3 log(M) rather than estimating M^2 parameters. The authors identify extensions beyond generalized linear models and the exponential function as future directions, subject to stability and boundedness challenges.
6 Conclusions
- For bounded-degree networks, the required data are on the order of ρ^3 log(M), compared with M^2 parameters being estimated.The bounds depend on sparsity, maximum node degree, ambient dimension, and observation time.Source: §6 · p. 21
- The analysis covers generalized linear models; extending it to other model families is presented as possible future work.The authors also propose exploring functions g beyond the exponential, noting that stability and high-probability boundedness are key challenges.Source: §6 · p. 21
- Higher-order or autoregressive moving average models are proposed as directions that could better model real-world point processes.Source: §6 · p. 21
7 Appendix
The appendix assembles probability bounds and proof tools for the paper’s results. It addresses Poisson truncation and tails, martingale concentration, stationary-distribution arguments, and empirical-process symmetrization.
7.1 Supplementary Lemmas
- Supplementary lemmas bound Poisson truncation error under stated U conditions and provide Poisson-tail and martingale concentration tools.The truncation argument treats X′ as a Poisson distribution with its tail removed and shows its variance is close to λ under the stated conditions.Source: §7.1 · pp. 22–23
7.2.1 Part 1
- For the Poisson autoregression, a sufficiently large C makes the tail bound valid across all M×T observations.The stated threshold is C > max(e^νmax(2e −1), 4 + e^νmax).Source: §7.2.1 · pp. 25–26
7.2.2 Part 2
- The tail-count argument bounds the fraction of observations above U; when U exceeds the stated threshold, the event probability decays in M and T.Source: §7.2.2 · pp. 26–27
7.3 Proof of Lemma 2
- The stationary-distribution proof uses a coupling of chains to bound their total-variation distance.The proof also invokes the symmetry assumption A*=A*⊤.Source: §7.3 · p. 28
7.4 Empirical processes for martingale sequences
- For martingale sequences, sequential Rademacher complexity supplies a symmetrization tool for empirical-process arguments.The appendix defines the complexity using trees whose node labels depend on prior Rademacher signs.Source: §7.4 · p. 29