Lasso and probabilistic inequalities for multivariate point processes

Niels Richard Hansen; Patricia Reynaud-Bouret; Vincent Rivoirard

Source-linked AI summary

Lasso and probabilistic inequalities for multivariate point processes

Niels Richard Hansen, Patricia Reynaud-Bouret, Vincent Rivoirard

arXiv:1208.0570v2Aug 2, 2012math.ST

TL;DR

The paper addresses multivariate point-process estimation where conventional nonparametric approaches are computationally costly and require ad-hoc tuning. It proposes a convex Lasso method with data-driven weights, proving tuning robustness and strong simulation performance against adaptive Lasso.

Problem
Multivariate Hawkes estimation needs a computationally feasible alternative to nonconvex ℓ0 penalization with non-data-driven, ad-hoc tuning.
Method
The authors use a convex Lasso criterion with data-driven penalty weights derived from Bernstein-type concentration inequalities for multivariate point processes.
Results
The method outperforms adaptive Lasso in simulations, with γ = 1 recovering dependency groups and other nonzero components when recordings are sufficiently long.
Takeaways & Limitations
Choosing γ = 1 provides a practically recommended tuning value that is robust to recording duration, unlike adaptive Lasso.
Takeaways & Limitations
Adaptive tuning comparisons must account for non-symmetric counting-process martingales and cases where the empirical variance estimate can be zero.

Abstract

from arXiv · show

Due to its low computational cost, Lasso is an attractive regularization method for high-dimensional statistical settings. In this paper, we consider multivariate counting processes depending on an unknown function parameter to be estimated by linear combinations of a fixed dictionary. To select coefficients, we propose an adaptive $\ell_1$-penalization methodology, where data-driven weights of the penalty are derived from new Bernstein type inequalities for martingales. Oracle inequalities are established under assumptions on the Gram matrix of the dictionary. Nonasymptotic probabilistic results for multivariate Hawkes processes are proven, which allows us to check these assumptions by considering general dictionaries based on histograms, Fourier or wavelet bases. Motivated by problems of neuronal activity inference, we finally carry out a simulation study for multivariate Hawkes processes and compare our methodology with the adaptive Lasso procedure proposed by Zou in (J. Amer. Statist. Assoc. 101 (2006) 1418-1429). We observe an excellent behavior of our procedure. We rely on theoretical aspects for the essential question of tuning our methodology. Unlike adaptive Lasso of (J. Amer. Statist. Assoc. 101 (2006) 1418-1429), our tuning procedure is proven to be robust with respect to all the parameters of the problem, revealing its potential for concrete purposes, in particular in neuroscience.

1. Introduction

The paper develops adaptive Lasso estimation for function parameters in multivariate counting processes, with data-driven penalties supported by martingale inequalities and Gram-matrix conditions. It specializes the framework to Hawkes processes and evaluates practical tuning and estimation for neuronal-network inference.

Problem and framework: The method estimates and selects coefficients in intensity-function expansions for multivariate point processes using an ℓ1-penalized least-squares criterion.The framework allows the target function parameter to belong to a Hilbert space and uses linear combinations of a fixed dictionary.
Methodological contributions: Data-driven penalty weights are calibrated using observable martingale bounds derived from Bernstein-type inequalities.The paper emphasizes a sharp random observable upper bound rather than deterministic or unobservable bounds.
Models and theory: The general framework covers Poisson, Aalen multiplicative-intensity, and multivariate Hawkes processes, with asymptotics varying by model.For Poisson and Aalen examples asymptotics concern M, whereas for Hawkes processes M is fixed and recording duration T varies.
Models and theory: New Hawkes-process probability bounds allow explicit verification of the Gram-matrix condition with probability close to 1.The results control point counts and convergence in the ergodic theorem for multivariate Hawkes processes.
Empirical study: A simulation study evaluates implementability, tuning, and neuronal-network inference, including recovery of dependency groups and functional connectivity graphs.The study compares the proposed procedure with adaptive Lasso and uses a two-step ordinary-least-squares refinement after support selection.

2. Lasso estimate and oracle inequality

The paper estimates a function parameter through an ℓ1-penalized expansion over a dictionary and derives oracle risk bounds under a Gram-matrix condition. The bound balances approximation error against variance from martingale fluctuations, with data-driven weights calibrated by concentration inequalities.

Estimator: The estimator represents the target parameter as a linear combination of dictionary functions and minimizes a least-squares contrast with ℓ1 penalization.The resulting estimate is designed to combine sparsity with statistical guarantees.
Oracle condition: The oracle inequality requires a condition ensuring that the random Gram matrix is invertible and defines a valid norm on dictionary combinations.The procedure itself does not require knowing the condition's constant for implementation.
Oracle inequality: The risk bound combines an approximation term with a variance term controlling random fluctuations of empirical coefficients around their expectations.A sparse representation can reduce the variance contribution, while potentially increasing approximation error.
Oracle inequality: If the target is exactly represented by the dictionary, the oracle bound reduces to estimation errors for the representation's components.This gives a sharp control of the estimation error under the stated assumptions.
Data-driven calibration: Data-driven ℓ1 weights are derived from martingale concentration control, and the resulting oracle inequality holds with high probability when the required conditions are satisfied.The tuning parameter x trades off the probability of the event against the size of the penalty weights.
Penalty calibration: The quadratic penalty component can approach its probabilistically sharp form, but doing so increases the accompanying linear residual term.The trade-off is controlled through the choices of μ and ε.

3. Bernstein type inequalities for multivariate point processes

The paper develops Bernstein-type concentration inequalities for martingales associated with multivariate counting processes. Observable quadratic-variation estimates replace deterministic variance bounds, while boundedness and exponential-moment assumptions govern the result.

Main inequality: A Bernstein-type concentration inequality under boundedness assumptions supplies the penalty-weight calibration needed in the oracle inequality.The result is presented as both a probabilistic contribution and the basis for selecting the vector d.
Assumptions: The theorem applies to multivariate counting processes with predictable intensities and predictable integrands satisfying exponential-moment conditions.The assumptions control both first- and second-order exponential terms over the observation interval.
Bound structure: The concentration bound uses a variance-like quadratic variation together with a deterministic bound B on the integrand magnitude.For moderate x and sufficiently large observation duration, the leading term scales like √(2ρx).
Sharpness: The quadratic term is essentially unimprovable, although the exact sharp constant is reached only asymptotically and additive B-dependent terms remain.A peeling argument is used in the proof.
Observable normalization: Theorem 3 improves a deterministic upper bound by plugging in the unbiased observable estimate v̂ of the quadratic variation.This produces a random, data-dependent upper bound rather than relying solely on a nonsharp deterministic variance bound.
Scope and robustness: The proposed result avoids requiring conditional symmetry, which is unavailable for general counting processes and dictionaries.Alternative inequalities may require a positive lower bound on v̂, which fails when the process is empty; the theorem instead permits w = B^2x.

4. Applications to the Poisson and Aalen models

The general oracle framework is instantiated for Poisson and Aalen multiplicative-intensity models. Under model-specific assumptions, the Gram-matrix condition holds with high probability and the estimator adaptively balances approximation bias against variance.

Poisson model: The Poisson application considers M independent processes with a common intensity supported on [0,1] and an orthonormal dictionary.The Gram matrix simplifies under this setup, enabling direct application of the general theorem.
Poisson model: For the Poisson model, concentration of the total number of observed points yields high-probability control of the Gram-matrix condition.The argument uses the Poisson distribution and a logarithmic choice of the concentration parameter.
Poisson model: The resulting bound has a variance contribution corresponding, up to logarithmic factors, to the sum of component variance terms.The estimator adaptively achieves the best trade-off between approximation bias and variance, with the logarithmic factor as the price of adaptation.
Aalen model: The Aalen application uses an orthonormal dictionary on L2([0,1]^2) and assumes regularity conditions on covariate densities and the associated empirical norm.Lower bounds on covariate density and related conditions ensure norm equivalence.
Aalen model: The Aalen case is less intricate than Hawkes processes but requires more cumbersome control of the exceptional event because intensities depend on covariates and response variables.The paper notes that simpler assumptions are used to avoid unnecessary technical complications.
Aalen model: For the Aalen model, the Gram-matrix condition is verified with high probability under bounded covariate density and assumptions controlling the observation process.The resulting corollary provides the corresponding oracle bound with constants depending on the model parameters.
Comparison: In both applications, the asymptotic conclusions depend on dictionary conditions that are discussed further for more involved settings.The paper explicitly defers the detailed choice of dictionary and verification conditions to the Hawkes-process analysis.

5. Applications to the case of multivariate Hawkes process

The paper develops probabilistic controls and oracle inequalities for adaptive Lasso estimation in multivariate Hawkes processes. Under stationarity, spectral-radius, interaction, and dictionary conditions, these results support observable guarantees and explicit dictionary constructions.

Probabilistic results: New exponential and tail-control results for multivariate Hawkes processes provide nonasymptotic bounds for point counts and related functionals.The results generalize earlier univariate exponential controls and are used to control events needed by the estimation theory.
Hawkes-process assumptions: The analysis uses a Poisson cluster representation for nonnegative multivariate Hawkes interactions, with clusters finite almost surely when the spectral radius of Γ is strictly smaller than 1.This condition also yields a stationary Hawkes-process version through the cluster representation.
Oracle inequalities: The paper establishes oracle inequalities for multivariate Hawkes processes and links dictionary properties, especially orthonormality, to control of the Gram-matrix event.These results are developed under the main Hawkes assumptions and lead to explicit probability guarantees for selected penalty parameters.
Dictionary construction: For dictionaries built from an orthonormal family in L2([0,1]), the resulting multivariate dictionary is orthonormal and has cardinality M + KM^2.The construction allows histogram, Fourier, and wavelet-type bases to be incorporated into the Hawkes-process analysis.
Dictionary-specific guarantees: The asymptotic event probability tends to 1 when α ≥1/2 for Fourier and histogram dictionaries and α ≥2/5 for compactly supported wavelets.The resulting oracle inequality has the usual bias-variance structure up to logarithmic terms, with asymptotics in the recording duration T.

6. Simulations for the multivariate Hawkes process

Simulations of multivariate Hawkes processes compare the proposed Bernstein-calibrated procedure with adaptive Lasso across network structures, recording durations, and interaction types. The proposed method generally recovers dependency groups, spontaneous rates, interaction functions, and coefficients, with the two-step procedure often performing best for parameter estimation.

Support recovery: At T = 2, the proposed method correctly recovered the three dependency groups in 32% of eight-neuron simulations with γ = 1, rising to near 100% at T = 20.Adaptive Lasso required γ = 1000 for convincing dependency-group recovery, while smaller values produced poor estimates.
Support recovery: The proposed method optimally detected nonzero spontaneous rates across experiments and γ values, whereas adaptive Lasso missed some when T = 2.The comparison concerns the median number of nonzero spontaneous-rate estimates reported in Table 1.
Function estimation: For parameter estimation, the second OLS step improves results and stabilizes MSE, overcoming shrinkage that otherwise worsens for larger γ.For the proposed method, MSE increases with γ for B and A, while this pattern does not appear for the two-step procedures.
Function estimation: Reconstructions improve as T grows and with the second OLS step; for eight neurons at T = 20, BO gives the clearest estimation hierarchy and B the weakest.The reported ordering is BO best, followed by the other methods, with B worst; the two-step procedure also reduces shrinkage effects.
Conclusions: The recommended γ = 1 is robust to recording duration, and the proposed method outperforms adaptive Lasso while recovering negative interactions in the inhibition experiment.The authors report recovery of dependency groups, spontaneous rates, nonzero functions, and, with sufficiently large T, nonzero coefficients.

7. Proofs

This section introduces the paper’s proofs and fixes the convention that C denotes a line-dependent constant.

The section is devoted to proving the paper’s results.
Throughout the proofs, C denotes a constant whose value may change from line to line.
The proof convention allows C to take different values in different lines.

7.1. Proof of Theorem 1

The proof of Theorem 1 combines a standard inequality, an assumption on the Gram matrix, the triangle inequality, and an elementary quadratic bound.

The proof begins from the relation obtained using equation (2.5).
The Gram-matrix assumption and the triangle inequality for the T-norm produce the main bound.
The inequality 2xy ≤ αx^2 + α^-1y^2 is applied for α ∈ (0,1).
The theorem follows by choosing an arbitrary absolute value for α ∈ (0,1).

7.2. Proof of Theorem 2

The proof of Theorem 2 constructs a stopped predictable process, applies martingale concentration, and bounds the complementary event to obtain the result.

The proof defines a stopping time τ′ and a predictable process H.
Theorem 3 is applied to H with τ = T and B = Bϕ after verifying its integrability conditions.
On ΩV,B, the stopping time satisfies τ′ ≥ T, allowing the argument to proceed for all t ≤ T.
The proof rewrites the martingale expression and applies Theorem 1 on the intersection of the required events.

7.3. Proof of Theorem 3

The proof of Theorem 3 derives martingale concentration bounds, replaces an unobservable quadratic characteristic with an estimator, and uses peeling to obtain the final inequality.

The proof normalizes B to 1 and fixes ξ ∈ (0,3) before applying exponential-martingale arguments.
The concentration event controls the martingale term ξH • (N − Λ) up to the stopping time.
The bound on φ(ξ) yields an explicit expression involving 2(1 − ξ/3)v + ξ^-1x.
A geometric peeling construction handles the random quadratic characteristic H^2 • Λτ.
The resulting proposition applies to multivariate counting processes with predictable intensities and predictable H.
The proof replaces H^2 • Λτ with the observable quadratic variation H^2 • Nτ through an additional martingale argument.
The final bound is expressed using the estimator V̂μ under the corresponding event conditions.

7.4. Proofs of the probabilistic results for Hawkes processes

The proofs establish local contraction and moment bounds for branching-process representations of Hawkes clusters, then use these properties to derive probabilistic controls under stationarity and light-tailed assumptions.

Branching-process representation: The log-Laplace transform of generation counts is represented recursively through iterates of g(θ) = θ + φ(θ).The map φ is defined from the offspring distribution and depends only on the mean offspring matrix Γ.
Branching-process representation: A spectral radius below one makes φ a contraction near zero, keeping the iterates g◦n(θ) inside a suitable neighborhood.Continuity of Dφ provides a radius r and contraction constant C < 1 under an appropriate norm.
Moment control: The contraction argument requires only that φ be defined near zero, corresponding to sufficiently light-tailed offspring distributions rather than the explicit Poisson formula.For Poisson offspring distributions, the explicit formula is available but is not needed in this proof.
Moment control: Cluster representations and exponential moments yield finite Laplace-transform bounds for point counts in bounded intervals.The proof decomposes counts by ancestral types and uses independence of ancestral counts and cluster sizes.
Probabilistic controls: The probability of the exceptional event is bounded by □α,A,f0T exp(−□α,A,f0 Ñ), and choosing Ñ = C3 log(T) gives a polynomial bound in T.The bound holds for every positive choice of Ñ, allowing logarithmic growth to control the exceptional probability.
Probabilistic controls: A likelihood-process argument under a Poisson reference measure produces a bound under stationarity when the local point count has a finite (1 + ε)-moment.The likelihood process compares the original process with independent unit-rate Poisson components after time zero.

7.5. Proofs of the results of Sections 4.2 and 5.2

These proofs control Gram-matrix fluctuations and martingale terms using exponential point-count bounds, Bernstein inequalities, and the dictionary’s norm structure.

Section 4.2: The Gram matrix is controlled by combining its population lower bound with concentration bounds for every pair of dictionary elements.For an orthonormal dictionary, the population quadratic form is bounded below by Tζ times the coefficient ℓ2 norm.
Section 4.2: The proof applies Cauchy–Schwarz and elementary exponential inequalities before choosing tuning parameters to obtain the required concentration result.The argument also uses the finite size of the dictionary and exponential moments of the local point count.
Section 5.2: The nonasymptotic result follows from Theorem 2 after controlling the martingale ψ(ϕ)^2 • (N − Λ)T on a suitable high-probability event.On the event, the auxiliary process H matches ψ(ϕ)^2 and its compensator term matches ψ(ϕ)^4 • ΛT.
Section 5.2: With x = α log(T), the simultaneous probability bound becomes 1 − (M + KM^2)T^−α for all ϕ ∈ Φ.The result is obtained on the event controlling the relevant count and variation quantities.
Section 5.2: Taking N = log^2(T) and applying Corollary 3 completes the concentration argument with the constant choice specified in Proposition 5.The final parameter choices are made to satisfy the proposition’s required bounds.