Source-linked AI summary
Causal Recurrent Variational Autoencoder for Medical Time Series Generation
Hongming Li, Shujian Yu, Jose Principe
TL;DR
Time-series generators need to represent temporal dynamics and latent factors, while causal discovery from time series is a separate goal. CR-VAE integrates Granger-causal structure into a recurrent VAE generator, and the authors report competitive generation and causal-discovery results. They also identify assumptions that bound the method’s scope.
Problem
Many time-series models capture conditional dynamics but are deterministic, limiting their ability to infer latent factors and generate series by sampling from a tractable latent distribution.
Source: Introduction · p. 1Method
CR-VAE uses a recurrent VAE with a multi-head decoder and sparsity on decoder weights to learn a Granger-causal adjacency matrix that structures generation.
Source: Introduction · p. 1Results
CR-VAE achieves competitive performance in generation and causal discovery, and has smaller maximum mean discrepancy values and prediction mean square errors under train-on-synthetic, test-on-real evaluation.
Source: Introduction · p. 2, Conclusion · p. 7Takeaways & Limitations
The learned Granger-causal graph makes the generation process more transparent by encoding causal relations between variables.
Source: Introduction · p. 1, Conclusion · p. 7The model assumes no unmeasured confounders, and its isotropic Gaussian latent-factor assumption limits generative capability.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
We propose causal recurrent variational autoencoder (CR-VAE), a novel generative model that is able to learn a Granger causal graph from a multivariate time series x and incorporates the underlying causal mechanism into its data generation process. Distinct to the classical recurrent VAEs, our CR-VAE uses a multi-head decoder, in which the $p$-th head is responsible for generating the $p$-th dimension of $\mathbf{x}$ (i.e., $\mathbf{x}^p$). By imposing a sparsity-inducing penalty on the weights (of the decoder) and encouraging specific sets of weights to be zero, our CR-VAE learns a sparse adjacency matrix that encodes causal relations between all pairs of variables. Thanks to this causal matrix, our decoder strictly obeys the underlying principles of Granger causality, thereby making the data generating process transparent. We develop a two-stage approach to train the overall objective. Empirically, we evaluate the behavior of our model in synthetic data and two real-world human brain datasets involving, respectively, the electroencephalography (EEG) signals and the functional magnetic resonance imaging (fMRI) data. Our model consistently outperforms state-of-the-art time series generative models both qualitatively and quantitatively. Moreover, it also discovers a faithful causal graph with similar or improved accuracy over existing Granger causality-based causal inference methods. Code of CR-VAE is publicly available at https://github.com/hongmingli1995/CR-VAE.
Introduction
The paper addresses the need for time-series generators that capture temporal dynamics and latent factors while also supporting causal discovery. CR-VAE combines recurrent VAE generation with a Granger-causal graph, and the authors report competitive results on generation and causal discovery.
Introduction
- Many predictive models capture conditional dynamics but are deterministic, so they cannot infer latent factors and generate new series by sampling from a tractable latent distribution.A good generator is expected to model both the joint distribution and temporal transition dynamics.Source: Introduction · p. 1
- CR-VAE assigns each decoder head one variable and sparsifies decoder weights to learn a causal adjacency matrix that structures generation according to Granger causality.The model also includes an error-compensation module for instantaneous influence.Source: Introduction · pp. 1–2
- CR-VAE achieves competitive performance in both time-series generation and Granger-causal discovery across synthetic sequences and real-world medical time series.The evaluation compares synthetic and real data distribution closeness and causal graphs against state-of-the-art approaches.Source: Introduction · p. 2
Time Series Generative Models
- GAN-based generators model the joint time-series distribution but may omit transition dynamics, while TimeGAN estimates conditional density in a latent space.Earlier VAE approaches are less investigated; some encode future information, and TimeVAE adds decoder blocks with practice-sensitive hyperparameters.Source: Time Series Generative Models · p. 2
- CR-VAE is presented as combining Granger-causal discovery, explicit conditional-density modeling, and a generation process designed to obey Granger-causality principles.The authors describe these as properties distinguishing it from the approaches discussed.Source: Time Series Generative Models · p. 2
Causal Discovery of Time Series
- Time-series causal discovery includes information-theoretic and Granger-based approaches, but the cited discussion notes that existing causal-inference methods cannot also generate time series.Transfer entropy can be difficult to estimate in high-dimensional settings and cannot quantify instantaneous causality; neural approaches use sparsity or attention-based architectures.Source: Causal Discovery of Time Series · pp. 2–3
Causal Recurrent Variational Autoencoder
CR-VAE jointly models time-series dynamics and a Granger causal graph, using recurrent, variable-specific decoders constrained by the learned graph. Its training combines variational reconstruction with sparse graph learning and separates graph estimation from later generation-focused refinement.
Problem Formulation and Objectives
- The formulation seeks a matching joint time-series distribution and learns each variable’s function alongside an adjacency matrix encoding its Granger causes.The model represents each variable using past observations of its parent variables, with jointly independent noise terms.Source: Problem Formulation and Objectives · p. 3
Methodology
- CR-VAE encodes an earlier segment to predict or reconstruct the next segment, preventing future information from entering the recurrent decoder before generation.Its decoder uses multiple heads, each approximating the function for one variable; the row of the estimated adjacency matrix for that variable identifies its causes.Source: Methodology · pp. 3–4
CR-VAE Loss Function
- The objective combines reconstruction error, a latent-space KL term, and a sparsity penalty on the estimated adjacency matrix; a separate VAE objective trains the innovation-compensation network.The innovation term has no specific distributional assumption, and the compensation network’s update does not affect the estimated causal matrix.Source: Methodology · p. 3, CR-VAE Loss Function · p. 4
CR-VAE Learning and Optimization
- Because the ℓ1 sparsity penalty can reduce generation performance, two-stage training first estimates a sparse graph, fixes its zero entries, then uses SGD to improve generation.The authors use ℓ1 instead of the difficult-to-optimize ℓ0 penalty; ISTA thresholding can produce exact zero weights.Source: CR-VAE Learning and Optimization · p. 5
Experiments
The experiments test CR-VAE’s components, then evaluate causal discovery and synthetic time-series generation on synthetic and medical datasets. CR-VAE performs competitively across both tasks, with the reported comparisons favoring it over TCDF and generation baselines except for VRAE on fMRI.
Experiments
- The experiments are framed as tests of CR-VAE’s components and comparisons against state-of-the-art methods on four time-series datasets.The study assesses both causal discovery and synthetic time-series generation.Source: Experiments · p. 5, Experiments on Different Data · p. 6
Linear Autoregressive Process
- The component evaluation uses a 10-dimensional synthetic linear autoregressive process with maximum lag 3.The training pipeline updates decoder input weights with proximal gradient descent, prunes zero edges, then continues training the remaining parameters.Source: Linear Autoregressive Process · p. 5
Unidirectional Inputs: Don’t Peep on the Future.
- The future-peeking non-unidirectional CR-VAE fails to identify causal directions between most pairs, while CR-VAE recovers most true relations.The comparison uses the synthetic linear autoregressive process; the baseline encoder receives later sequence values than the unidirectional version.Source: Unidirectional Inputs: Don’t Peep on… · p. 5, Linear Autoregressive Process · p. 5, Figure 2 · p. 6
Error-Compensation Network.
- Without error compensation, CR-VAE’s generated samples quickly converge to values near zero despite capturing the autoregressive transition dynamics.The degraded model omits ε_t, which the passage identifies as the reason the dynamics are ignored in generation.Source: Error-Compensation Network · pp. 5–6, Figure 3 · p. 6
Experiments on Different Data
- The evaluation compares CR-VAE with four Granger-causal discovery approaches and three time-series generation baselines across four datasets.The datasets include Hénon maps, Lorenz-96, fMRI, and EEG; the EEG data have no ground-truth causal relations.Source: Causal Discovery Evaluations · p. 6, Experiments on Different Data · p. 6, Time Series Generation Evaluation · p. 7
Causal Discovery Evaluations
- CR-VAE outperforms TCDF on all evaluated datasets and achieves performance similar to NGC in causal discovery.The comparison reports AUROC results; both CR-VAE and NGC apply an ℓ1 sparsity penalty to network weights.Source: Causal Discovery Evaluations · p. 6, Table 1 · p. 7
Time Series Generation Evaluation
- CR-VAE consistently generates higher-quality synthetic data than baselines, though VRAE slightly outperforms it on fMRI.In t-SNE projections, CR-VAE overlaps markedly better with real data than TimeGAN and slightly better than VRAE; fMRI samples are nearly indistinguishable.Source: Time Series Generation Evaluation · p. 7, Table 2 · p. 7, Figure 5 · p. 7
Conclusion
CR-VAE integrates Granger causality into a recurrent VAE and is evaluated on synthetic systems and benchmark medical datasets. The conclusion reports improved TSTR metrics, while noting assumptions about confounders and latent factors.
Conclusion
- CR-VAE integrates Granger causality into a recurrent VAE, supporting discovery of pairwise causal relations and a more transparent generation process.Source: Conclusion · p. 7
- Across two synthetic dynamic systems and two benchmark medical datasets, CR-VAE has smaller maximum mean discrepancy values and prediction mean square errors under TSTR.Source: Conclusion · p. 7
- The model assumes no unmeasured confounders, and its isotropic Gaussian latent-factor assumption limits generative capability.Source: Conclusion · p. 7
Supplementary Material
The supplementary material is organized around dataset and baseline-method details, followed by additional results and an ablation study.
Supplementary Material
- The document is supplementary material for the “Causal Recurrent Variational Autoencoder for Medical Time Series Generation” manuscript and is organized into topics and sections.Source: Supplementary Material · p. 8
2. Details of Experimental Setting
The experimental-setting supplement covers dataset details and descriptions of traditional and neural network-based Granger-causality methods.
3. Additional Experimental Results
The supplementary results include causal-graph summaries, generated-series comparisons, TSTR prediction results, PCA visualizations, and an ablation of two-stage training.
3. Additional Experimental Results
- Additional results include adjacency matrices of Granger causal summary graphs and synthetic time series generated by different methods.Source: §3 · p. 8
- The supplement also presents TSTR prediction results and PCA visualizations.Source: §3 · p. 8
- An additional ablation study examines the two-stage training strategy.Source: §3 · p. 8
4. Minimal Structure of CR-VAE in PyTorch and Code Link
The paper distinguishes three time-series causal graph representations and places its Granger causal graph between window-level and summary graphs. It situates CR-VAE among Granger-causality, constraint-based, noise-based, and score-based approaches.
More on Related Works
- Window graphs encode causal influence across a fixed lag window, while summary graphs relate variables without specifying time instants.The paper notes that window graphs are harder to estimate because they require identifying exact causal time instants.Source: More on Related Works · p. 8
- The Granger causal graph represents whether a variable’s past causes another variable’s present, placing it between window and summary graphs.It separates past observations from present values.Source: More on Related Works · p. 8
- Time-series causal discovery includes Granger-causality and constraint-based approaches, alongside noise-based methods such as TiMINo and score-based methods such as DYNOTEARS.CR-VAE belongs to the Granger-causality-based approach.Source: More on Related Works · pp. 8–9
Details of Experiment Setting
The experiments use four datasets and compare causal-discovery and generation methods under specified tuning procedures. Generation quality is assessed with distributional and prediction metrics.
More Details of Dataset
- The experiments use four datasets and more than 2,000 samples per dataset; fMRI uses 2,048 cropped clips, while EEG uses the 12 deep-structure contacts without causal ground truth.The Hénon data have lag 2 and six variables; the fMRI true lag is unavailable.Source: More Details of Dataset · p. 9, Table 3 · p. 9
Details of Traditional Granger Causality
- KGC and transfer entropy are tuned by grid search, with the main-manuscript settings selected for highest AUROC.Transfer entropy is estimated from kernel-based Rényi entropy quantities.Source: Details of Traditional Granger Causality · p. 9
Details of Neural Network-based Approaches of Granger Causality
- Neural causal-discovery methods are tuned to assumed sparsity levels, then other hyperparameters are selected by minimizing loss without the sparsity penalty.For Hénon, the assumed causal sparsity ranges from 20% to 40%.Source: Details of Neural Network-based… · p. 10, Table 5 · p. 10
Details of Generation
- TimeGAN, VRAE, and CR-VAE are compared as generation baselines, with hyperparameters searched to minimize maximum mean discrepancy (MMD).TimeGAN is GAN-based; VRAE is a recurrent VAE extension.Source: Details of Generation · p. 10
Evaluation Metrics of Generation
- MMD measures distance between real and synthetic distributions, while TSTR trains a predictor on synthetic data and evaluates real-data predictions by RMSE.MMD averages Gaussian-kernel bandwidths from 0.01 to 100; TSTR uses next-step prediction.Source: Evaluation Metrics of Generation · pp. 10–11
Additional Experimental Results
Additional results compare discovered causal matrices, generated sequences, predictions, and PCA projections, and examine the two-stage training strategy. Reported findings favor CR-VAE on several comparisons while warning that PCA can misrepresent generation quality.
Adjacency Matrices of Granger Causal Summary Graphs
- As illustrated in Figure 7, CR-VAE outperforms most of the baselines and achieves competitive results among neural network-based methods.False positive or negative elements are highlighted by red squares.Source: Adjacency Matrices of Granger Causal… · p. 11
Synthetic Time Series by Different Methods
- CR-VAE’s generated sequences resemble VRAE’s and outperform TimeGAN’s across the datasets shown.Figure 8 displays 10 generated sequences per subplot to show time-series texture.Source: Synthetic Time Series by Different… · p. 12, Figure 8 · p. 12
Prediction Results of TSTR
- On Hénon, Lorenz 96, and EEG, CR-VAE’s predicted curves overlap the original curves more closely.Figure 9 shows 20-length sequences predicted stepwise on real data.Source: Prediction Results of TSTR · p. 13, Figure 9 · p. 13
PCA Visualization
- PCA projections are inconsistent with t-SNE, MMD, and TSTR, so the authors judge PCA unsuitable for evaluating time-series generation quality.The Hénon TimeGAN samples cluster at a few fixed values, unlike the other PCA plots.Source: PCA Visualization · p. 13, Figure 10 · p. 14
Ablation Study of the Two-Stage Learning Strategy
- Stage II training significantly improves generation quality after sparse weights are fixed, with the fMRI learning curve dropping after Stage II begins.Figure 11 compares fMRI generation with and without Stage II using t-SNE projections.Source: Ablation Study of the Two-Stage… · p. 14, Figure 11 · p. 15
Minimal Structure of CR-VAE in PyTorch and Code Link
The minimal PyTorch structure uses one encoder GRU and a set of decoder GRUs linked to the causal connection matrix. The code is publicly available.
Minimal Structure of CR-VAE in PyTorch and Code Link
- PyTorch code for CR-VAE is available at the project’s GitHub repository.The repository URL is https://github.com/hongmingli1995/CR-VAE.Source: Minimal Structure of CR-VAE in PyTorch… · p. 15
- The implementation uses one encoder GRU and one decoder GRU per time series, with the connection matrix determining each decoder’s inputs.The connection matrix is also used to prune zero edges during Stage II learning.Source: Minimal Structure of CR-VAE in PyTorch… · p. 15
- In generation mode, the model iteratively produces a sequence from an initial hidden state and zero input.The code returns the generated sequence after stepping through the decoder GRUs.Source: Minimal Structure of CR-VAE in PyTorch… · p. 16