Source-linked AI summary
Neural Granger Causality
Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaie, Emily Fox
TL;DR
Classical Granger-causality methods often assume linear dynamics, while real-world interactions can be nonlinear and difficult to estimate with model-free methods in high-dimensional settings. The paper uses structured-sparse neural models to recover interpretable nonlinear interactions, reporting better DREAM3 performance than existing approaches and interpretable structure in human motion data.
Problem
Linear model assumptions can miss nonlinear dependencies, while model-free estimators have high variance, require substantial data, and face dimensionality problems.
Source: §1 · p. 1Method
The paper models each output with a component-wise MLP or RNN and applies sparsity penalties to input weights to select nonlinear Granger-causal relationships.
Source: §1 · p. 2, §3.2.1 · p. 4, §3.3 · p. 5Results
The cMLP and cLSTM methods outperform existing Granger-causality approaches on DREAM3 and reveal interpretable structure in human motion data.
Source: §1 · p. 2, §8 · p. 10, §9 · p. 11Takeaways & Limitations
Structured sparsity supports interpretable interaction recovery and regularization when neural models are applied to limited-data settings.
Source: §1 · p. 2Zero outgoing input weights are sufficient but not necessary for Granger non-causality, and the penalties bias the models toward representing non-causality with zeros.
Source: §5 · p. 7
Abstract
from arXiv · showhide
While most classical approaches to Granger causality detection assume linear dynamics, many interactions in real-world applications, like neuroscience and genomics, are inherently nonlinear. In these cases, using linear models may lead to inconsistent estimation of Granger causal interactions. We propose a class of nonlinear methods by applying structured multilayer perceptrons (MLPs) or recurrent neural networks (RNNs) combined with sparsity-inducing penalties on the weights. By encouraging specific sets of weights to be zero--in particular, through the use of convex group-lasso penalties--we can extract the Granger causal structure. To further contrast with traditional approaches, our framework naturally enables us to efficiently capture long-range dependencies between series either via our RNNs or through an automatic lag selection in the MLP. We show that our neural Granger causality methods outperform state-of-the-art nonlinear Granger causality methods on the DREAM3 challenge data. This data consists of nonlinear gene expression and regulation time courses with only a limited number of time points. The successes we show in this challenging dataset provide a powerful example of how deep learning can be useful in cases that go beyond prediction on large datasets. We likewise illustrate our methods in detecting nonlinear interactions in a human motion capture dataset.
1 INTRODUCTION
The paper develops structured-sparse neural models for interpretable nonlinear Granger causality, addressing the limits of linear models and the black-box, data-intensive character of neural forecasting models. It reports stronger DREAM3 benchmark performance and illustrates the approach on human motion data.
1 INTRODUCTION
- Model-free methods can detect nonlinear dependencies with minimal assumptions, but their high variance, data demands, and dimensionality burden limit their use in high-dimensional settings.Model-based methods may also fail when dependencies fall outside their assumed model class, as can occur with nonlinear relationships.Source: §1 · p. 1
- Neural forecasting models can represent complex nonlinear interactions but provide little structural interpretability and require many parameters when jointly modeling many series.The authors position their method around interpreting interactions rather than the prediction-focused successes typically associated with neural networks on large datasets.Source: §1 · p. 2
- Structured penalties on grouped neural-network weights select Granger-causal inputs while supporting interpretable nonlinear models in limited-data, high-dimensional settings.The models use component-wise architectures to disentangle how lagged inputs affect individual output series.Source: §1 · p. 2
- For cMLPs, structured penalties can select both nonlinear Granger connections and their relevant lags; cLSTMs use recurrence to model long-range dependencies without explicit lag selection.The paper states that both approaches can select causal input series regardless of the interaction lag when the true network is sparse.Source: §1 · p. 2
- The cMLP and cLSTM methods outperform a wide set of competitors across all five DREAM3 datasets, and cLSTM reveals interpretable structure in human motion data.The DREAM3 benchmark concerns gene regulatory network recovery; the motion-capture analysis examines interactions between body parts.Source: §1 · p. 2
- The framework generalizes structured penalties used in regression to neural-network structure learning, extending their use beyond predictive architecture selection.The cited concurrent work concerns sparse-input neural networks for independent-data regression and classification.Source: §1 · p. 2
2 LINEAR GRANGER CAUSALITY
The section describes linear VAR-based Granger causality through lag-specific coefficient matrices, then explains how group and hierarchical penalties encode causal and lag selection.
2 LINEAR GRANGER CAUSALITY
- The VAR setup represents each time point as a linear combination of a specified number of past lags.The process is described as a p-dimensional stationary time series observed at T time points.Source: §2 · p. 2
- In a VAR model, series j does not Granger-cause series i when the corresponding coefficients are zero across all lags.The lag-k coefficient matrix specifies how lag k affects future series values.Source: §2 · p. 2
- A group-lasso penalty groups each series pair’s coefficients across lags, shrinking them together; its tuning parameter controls the degree of group sparsity.The criterion identifies non-causal pairs when their grouped coefficients are shrunk to zero.Source: §2 · pp. 2–3
- A hierarchical penalty can automatically select the lag order for each Granger-causal interaction.Its tuning parameter controls the selected lag order.Source: §2 · pp. 2–3
3 MODELS FOR NEURAL GRANGER CAUSALITY
The models make nonlinear Granger-causality discovery interpretable by fitting component-wise neural networks and using structured sparsity to identify irrelevant series and lags. MLPs support explicit lag selection, while recurrent models capture long-range dependencies.
3.1 Adapting Neural Networks for Granger Causality
- Separate output-specific models address shared-network interpretability and lag-selection problems; Granger non-causality means the output function is invariant to that series’ past.The section identifies standard joint neural forecasting models as difficult to interpret and as assuming each output depends on the same past lags.Source: §3.1 · p. 3
3.2 Sparse Input MLPs
- cMLPs fit a separate MLP for each output, with a linear output decoder chosen for real-valued multivariate time series.The paper notes that logistic, softmax, or Poisson decoders could instead model binary, categorical, or positive-count series.Source: §3.2 · p. 3
3.2.1 Penalized Selection of Granger Causality in the cMLP
- Across cMLP input lags, group lasso shrinks each series’ weights jointly, while group sparse group lasso selects both causal series and relevant lags.The mixed penalty provides sparsity across series groups and within groups of lags.Source: §3.2.1 · p. 4
- Hierarchical cMLP penalties select causal series and lag order, penalizing higher lags more and allowing a large maximum lag without missing delayed links.These penalties combine model fitting and lag selection, which the paper also identifies as useful for nonlinear forecasting.Source: §3.2.1 · pp. 4–5
3.3 Sparse Input RNNs
- A component-wise RNN uses a separate recurrent model for each output; its LSTM cell and hidden states represent historical context.The formulation assumes a single-layer RNN for simplicity and can be generalized to more layers.Source: §3.3 · p. 5
3.3.1 Granger Causality Selection in LSTMs
- In cLSTMs, group lasso across input-weight columns selects causal series; zeroing a series’ column is a sufficient condition for non-causality.The group penalty gives the LSTM gate and cell-input matrices a shared column-sparsity pattern.Source: Figure 3 · p. 4, §3.3.1 · p. 5
4 OPTIMIZING THE PENALIZED OBJECTIVES
The cMLP and cLSTM objectives are optimized with proximal gradient methods that apply sparsity penalties to input weights. The cLSTM procedure uses full backpropagation through time for short replicated series and truncated backpropagation for longer ones.
4.1 Optimizing the Penalized cMLP Objective
- Proximal gradient descent produces exact zeros in cMLP input-weight columns, enabling Granger non-causality interpretation; line search can ensure convergence to a local minimum.The weights are initialized randomly, and the step size may be fixed or selected by line search.Source: §4.1 · pp. 5–6
- For cMLP, group sparse group lasso thresholds lag-specific weights before thresholding all input weights for each series; the hierarchical penalty thresholds nested groups from smallest to largest.Higher-level weights receive the identity proximal step because sparsity penalties apply only to input weights.Source: §4.1 · p. 6
- For cMLP, gradients use the full-data objective on the small datasets studied; proximal stochastic gradient descent is an option for larger datasets.The section also provides one-pass algorithms for computing the group sparse group lasso and hierarchical penalty proximal maps.Source: §4.1 · p. 6
4.2 Optimizing the Penalized cLSTM Objective
- The cLSTM objective is optimized with proximal gradient descent and full backpropagation through time for short replicated series, but truncated backpropagation for longer series.For longer series, batches are treated as independent realizations, so the gradients approximate those of the full cLSTM model.Source: §4.2 · p. 6
5 COMPARING CMLP AND CLSTM MODELS FOR GRANGER CAUSALITY
The cMLP can use a large specified maximum lag because its lag-selection strategy removes higher lags, while the cLSTM learns interaction memory without specifying a maximum lag. Their sparsity penalties also favor zero outgoing input weights as the representation of Granger non-causality.
5 COMPARING CMLP AND CLSTM MODELS FOR GRANGER CAUSALITY
- cMLP requires a maximum lag K, but its lag-selection strategy can automatically remove higher lags when K is set large.For a length T series, cMLP uses T − K data points, whereas cLSTM uses T − 1.Source: §5 · pp. 6–7
- cLSTM specifies no maximum lag and instead learns each interaction’s memory; its longer-range modeling may matter for short independent replicates.The authors suggest this ability may help explain cLSTM performance on DREAM3 data.Source: §5 · p. 7
- Zero outgoing input weights are sufficient but not necessary for Granger non-causality, and the penalties bias both models toward that representation.Other weight configurations could represent non-causality through exact cancellations.Source: §5 · p. 7
6 SIMULATION EXPERIMENTS
Simulations assess whether cMLP and cLSTM recover Granger-causal structure in linear VAR and nonlinear Lorenz-96 data. The results report strong recovery against neural baselines and show that the hierarchical cMLP penalty remains robust as the chosen input lag grows.
6.1 cMLP and cLSTM Simulation Comparison
- The cMLP and cLSTM accurately reconstruct the underlying Granger-causality graph in both linear VAR and nonlinear Lorenz-96 simulations.The Lorenz-96 experiments use p = 20 series and vary series length and forcing constant; VAR experiments include lag orders 1 and 2.Source: §6.1 · p. 7, §6.1.1 · p. 7, §6.1.2 · p. 8
6.1.1 Lorenz-96 Model
- Both cMLP and cLSTM outperform IMV-LSTM and LOO-LSTM by a wide margin on Lorenz-96 data; cMLP leads at both forcing levels, with a narrowing gap as T grows.Performance improves with T, and the cMLP and cLSTM 95% confidence intervals are below 1% AUROC, compared with 3–5% for IMV-LSTM.Source: §6.1.1 · p. 8, Table 2 · p. 8
6.1.2 VAR Model
- On VAR data, cMLP and cLSTM outperform both neural baselines as T increases; cMLP is more robust with less data, while cLSTM wins on several occasions at T = 500 or T = 1000.The authors attribute IMV-LSTM’s weak performance to its lack of design for Granger discovery and suggest LOO-LSTM suffers from low-data overfitting and retained signal among remaining series.Source: §6.1.2 · p. 8, Table 2 · p. 8
6.2 Quantitative Analysis of the Hierarchical Penalty
- On simulated Lorenz-96 data, the hierarchical cMLP penalty outperforms group and mixed penalties across K and stays roughly constant as K increases.Performance declines with increasing K for the group and mixed penalties, while the hierarchical penalty’s robustness means precise lag specification is unnecessary.Source: Table 3 · p. 8, §6.2 · p. 9
6.3 Qualitative Analysis of the Hierarchical Penalty
- For cMLP, at λ = 4.09e−4, lag order is overestimated for a few input series and some false-positive Granger links appear; at λ = 7.94e−4, lag selection is almost perfect and the graph is correctly estimated.Higher λ penalizes longer lags more strongly; the VAR model does not perform noticeably better, and cross-validation may select λ in practice.Source: §6.3 · p. 9
7 DREAM CHALLENGE
On DREAM3 gene-expression data, cLSTM and cMLP were compared with linear and nonlinear Granger-causality methods. The cLSTM led AUROC across all five datasets, while both neural methods performed better than previous approaches on AUPR.
7 DREAM CHALLENGE
- DREAM3 contains five simulated gene-regulation datasets with 100 series each and only 46 replicates sampled at 21 time points.The datasets include two E. Coli and three Yeast networks; the 46 replicates provide 966 time points in total.Source: §7 · p. 9
- For cMLP automatic lag selection, Figure 5 maps input series by row and lag by column, with brighter colors indicating larger input-weight norms.The comparison includes a single-output cMLP and a VAR model across penalty strengths, alongside the true lag interactions.Source: Figure 5 · p. 9
- The comparison covered LASSO, G1DBN, and nonlinear kernel method OKVAR, using AUROC and AUPR as performance measures.Both measures were computed by sweeping the regularization parameter λ over a range.Source: §7 · p. 9
- cLSTM outperformed all previous methods on AUROC across five DREAM3 datasets; both neural methods did much better than previous approaches on AUPR.cLSTM outperformed cMLP on AUPR in three datasets.Source: §7 · pp. 9–10, Figure 6 · p. 10
8 DEPENDENCIES IN HUMAN MOTION CAPTURE DATA
The authors apply cLSTM to motion-capture recordings to infer nonlinear dependencies between body regions. The learned graphs include intuitive directed interactions, including links between knees, along arms, and between hands and toes.
8 DEPENDENCIES IN HUMAN MOTION CAPTURE DATA
- The analysis targets potentially long-range, nonlinear dependencies between body regions during natural motion, rather than motion classification or segmentation.Source: §8 · p. 10
- The analysis used 54 joint-angle and body-position series from two subjects, covering 24 body regions over 2,024 time points.Some regions have multiple recorded motion angles.Source: §8 · p. 10
- The cLSTM graph weights are normalized outgoing-weight norms, with edges for multiple degrees of freedom averaged and weights of 0.01 or less thresholded.The model was fit to length-20 segments across a range of λ values.Source: §8 · p. 10
- Because individual behaviors provided extremely limited data, the authors learned interactions from data aggregated across the behavior collection.The collection includes jumping jacks, twists, arm circles, knee raises, squats, punching, toe touches, and running in place.Source: §8 · p. 10, Figure 8 · p. 11
- Even in the sparsest graph, the model learned directed links in both directions between the knees, links down each arm, and links between hands and toes.The authors relate the knee links to alternating movement and the hand-to-toe links to toe touches.Source: §8 · p. 11
9 CONCLUSION
The paper presents regularized cMLP and cLSTM models for nonlinear Granger-causality selection, with separate networks for output components and sparsity-promoting penalties. It reports strong DREAM3 results and interpretable motion-capture structure, while pointing to further work on architectures and identifiability.
9 CONCLUSION
- The proposed framework uses separate networks for output components and sparsity-promoting penalties on incoming weights to select nonlinear Granger-causal relationships.It applies component-wise MLP and LSTM architectures to disentangle input-past effects on output futures.Source: §9 · p. 11
- The methods outperform existing Granger-causality approaches on DREAM3 and yield interpretable structure on human motion-capture data.Source: §9 · p. 11
- Future work could explore tree-structured penalties and more powerful architectures, including clockwork RNNs and dilated causal convolutions.The authors also identify further theoretical work on Granger non-causality identifiability in more complex network models as important.Source: §9 · p. 11
- A proposed alternative would learn hidden representations for each input series and model each output as a sparse nonlinear combination across all series’ hidden states.This architecture combines aspects of the cMLP and cLSTM approaches and permits shared hidden representations across output tasks.Source: §9 · p. 11
APPENDIX A MODEL ABLATIONS
The appendix examines hidden-unit counts and optimization choices on Lorenz-96 data. Performance was robust with few hidden units and improved with larger counts, while three optimizers gave similar results in the tested setting.
APPENDIX A MODEL ABLATIONS
- Both cMLP and cLSTM were robust with few hidden units, and performance improved as hidden-unit count increased.The comparison used H values from 5 to 100 on Lorenz-96 data with T = 1000.Source: Appendix A · pp. 11–12
- A combined architecture would learn an RNN hidden representation for each series, then use sparse cMLPs to predict each output from those representations.The proposed design would share hidden-feature information while retaining interpretable structure between series’ hidden states and outputs.Source: Figure 10 · p. 12
- Adam, ISTA, and GIST produced similar AUROC results for cMLP with H = 10 on Lorenz-96 at T = 1000.The optimizer comparison covered five initializations and forcing constants F ∈ (10, 40), but not lower T or cLSTM settings.Source: Appendix A · p. 12
- ISTA was used in the remaining experiments because it yields exact zeros and is more efficient than GIST.Adam is fastest but requires a thresholding parameter; GIST is slower but guarantees convergence to a local minimum and is less learning-rate-sensitive.Source: Appendix A · p. 12
APPENDIX B BASELINE METHODS
The appendix describes attention-based and leave-one-out LSTM baselines for inferring Granger relationships from input importance, evaluated using AUROC.
APPENDIX B BASELINE METHODS
- The Lorenz-96 tables compare AUROC across cMLP and cLSTM hidden-unit counts in one run, and across cMLP optimizers over five initializations with 95% confidence intervals.Source: Table 4 · p. 12, Table 5 · p. 12
- IMV-LSTM infers Granger relationships from mean attention values for each input series, thresholding them to trace an ROC curve and calculate AUROC.Separate IMV-Full models predict each series using all series as inputs.Source: Appendix B · p. 12
- LOO-LSTM estimates input importance by measuring the loss increase when each input series is withheld from a model predicting another series.The resulting loss-difference matrix is thresholded to trace an ROC curve and calculate AUROC.Source: Appendix B · p. 12