PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Spatio-Temporal Point Processes

Ricky T. Q. Chen, Brandon Amos, Maximilian Nickel

arXiv:2011.04583v3cs.LG

TL;DR

Modeling continuous-space, continuous-time events remains difficult because general STPP likelihoods require multivariate integrals and existing neural methods can restrict spatial distributions. The paper combines Neural ODEs with Jump and Attentive CNFs to model event-history-conditioned spatial distributions, achieving state-of-the-art performance on spatio-temporal datasets.

  • Problem

    General STPP likelihoods require multivariate integrals, while existing neural temporal point processes can still rely on likelihood approximations and restricted spatial distributions.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The approach combines Neural ODEs, Neural Jump SDE ideas, and continuous-time normalizing flows, using Jump and Attentive CNF architectures to condition spatial distributions on event history.

    Source: §1 · pp. 1–2, §3 · p. 4

  • Results

    The authors report state-of-the-art performance on spatio-temporal datasets collected from a wide range of domains.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Jump and Attentive CNFs achieve better log-likelihoods than the Time-varying CNF, suggesting that prediction in these data sets benefits from modeling event-history dependence.

    Source: §5 · p. 9

    The Jump CNF requires sequential ODE restarts, so its number of initial value problems grows linearly with event-history size and incurs substantial cost for large histories.

    Source: §3 · p. 5

Abstract

from arXiv · show

We propose a new class of parameterizations for spatio-temporal point processes which leverage Neural ODEs as a computational method and enable flexible, high-fidelity models of discrete events that are localized in continuous time and space. Central to our approach is a combination of continuous-time neural networks with two novel neural architectures, i.e., Jump and Attentive Continuous-time Normalizing Flows. This approach allows us to learn complex distributions for both the spatial and temporal domain and to condition non-trivially on the observed event history. We validate our models on data sets from a wide variety of contexts such as seismology, epidemiology, urban mobility, and neuroscience.

1 INTRODUCTION

Spatio-temporal point processes need high-fidelity models of events with joint spatial and temporal dependencies, but computational constraints have limited existing parameterizations. The paper proposes Neural ODE-based models that combine flexible spatial distributions with conditioning on event history.

1 INTRODUCTION

  • Computational constraints make high-fidelity joint spatial and temporal modeling difficult: general STPP likelihoods require multivariate integrals, prompting approximations and restrictions that can lose information or mis-specify models.Existing neural temporal point processes remain limited by likelihood approximations or restricted spatial distributions.Source: §1 · p. 1
  • Figure 1 depicts spatial distributions that update after an event and gradually return to the marginal distribution during periods without observations.Color denotes p(x|t).Source: Figure 1 · p. 1
  • Neural ODEs combined with Neural Jump SDE and continuous-time normalizing-flow ideas yield flexible, high-fidelity STPP parameterizations with exact likelihoods and history-conditioned spatial distributions.The proposed CNF-based architectures use discontinuous jumps or self-attention to condition spatial distributions on event history.Source: §1 · pp. 1–2
  • The authors construct five pre-processed data sets for benchmarking spatio-temporal event models.Source: §1 · p. 2

2 BACKGROUND

The background defines spatio-temporal point processes through event histories and conditional intensities, then explains why likelihood computation is challenging. It introduces continuous-time normalizing flows as a Neural ODE-based way to represent smoothly changing distributions with tractable probabilities.

2 BACKGROUND

  • An STPP models event times and locations, with a random event count; its conditional intensity describes instantaneous event likelihood given prior history, and the joint log-likelihood scores an observed history over [0,T].The shorthand λ*(t,x) denotes intensity conditional on the history.Source: §2 · p. 2
  • Maximum-likelihood training for general STPPs is difficult because the log-likelihood requires a multivariate integral; prior work often uses exponential-kernel KDEs with known antiderivatives.Source: §2 · p. 2
  • Normalizing flows transform a simpler base distribution through an invertible mapping, enabling exact computation of the transformed density without unknown normalization constants.The paper focuses on continuous-time flows parameterized by Neural ODEs.Source: §2 · p. 2
  • A Neural ODE evolves base-distribution samples along trajectories, producing distributions that change smoothly over time and can model spatial events.The ODE function can be parameterized by a Lipschitz-continuous neural network.Source: §2 · p. 3
  • Computing the evolving density requires solving the state and log-density equations together; the Jacobian trace can be estimated with Hutchinson’s method, and reverse-time adjoint ODEs give constant-memory gradients.The trace estimate uses vector-Jacobian products; even a single Monte Carlo sample supports stochastic-gradient training.Source: §2 · p. 3

3 NEURAL SPATIO-TEMPORAL POINT PROCESSES

The method combines continuous-time hidden-state dynamics with normalizing flows to model event timing and spatial distributions while retaining exact likelihood computation. It offers time-varying, event-jump, and attention-based spatial models, with an ablation showing improved Attentive CNF training from lower-variance estimates.

Neural Spatio-Temporal Point Processes

  • The framework separates temporal ground intensity from conditional spatial density, enabling exact likelihood computation for flexible spatio-temporal models.The two terms are the temporal log-likelihood and spatial log-likelihood.Source: §3 · p. 3

Latent Dynamics and Ground Intensity

  • A continuous-time hidden state summarizes event history, evolves between events, and jumps at observations to capture abrupt changes that may shape future dynamics.A softplus output ensures the ground intensity is positive.Source: §3 · p. 4

Time-varying CNF

  • The Time-varying CNF models spatial distributions as a function of event time and supports parallel likelihood evaluation, but cannot capture spatial propagation from prior events.It suits data with temporal patterns and little spatial dependence on previous events.Source: §3 · pp. 4–5

Jump CNF

  • Jump CNF conditions continuous flow and event-triggered distribution updates on the hidden state, allowing spatial distributions to depend on observed history.Because updates are sequential, solver restarts make computation costly as the number of historical events grows.Source: §3 · p. 5

Attentive CNF

  • Attentive CNF uses attention over prior-event sampling paths to condition spatial distributions while solving event trajectories in parallel.Its attention-based conditioning avoids compressing all history into the hidden state at each event.Source: §3 · pp. 5–6

Low-variance Log-likelihood Estimation

  • Detaching cross-event attention gradients yields a surrogate Jacobian with unchanged diagonal elements and zeros outside the block diagonal, lowering trace-estimator variance.The ablation reports faster convergence and better-converged Attentive CNFs with lower-variance estimates.Source: §3 · p. 6
  • On PINWHEEL, lower-variance estimates and gradients led to faster convergence and better-converged Attentive CNFs.The ablation is shown in Figure 3.Source: Figure 3 · p. 6, §3 · p. 6

4 RELATED WORK

Prior work on neural temporal point processes and continuous normalizing flows provides relevant components, but the paper differs in its exact-likelihood spatio-temporal modeling and its alignment of flow time with observation time.

Neural Temporal Point Processes

  • Neural TPP approaches such as RMTPPs and Neural Hawkes Processes use recurrent history states but approximate likelihoods with Monte Carlo, while Neural Jump SDEs support exact likelihoods but are not suited to complex continuous mark distributions.The paper builds on Neural Jump SDE ideas for ground intensity and uses flows for spatial distributions.Source: §4 · p. 7

Continuous Normalizing Flows

  • Unlike prior CNF applications that do not align flow time with observation time, this work trains on arbitrary observation times within a joint spatio-temporal point-process model.The paper notes that Riemannian CNFs could extend the approach to spherical domains.Source: §4 · p. 7

5 EXPERIMENTS

Experiments across varied spatio-temporal data sets show stronger likelihood performance from history-conditioned CNFs than simpler spatial baselines. Results also highlight the relationship between spatial modeling and temporal fit, and differences between the two CNF variants.

5 EXPERIMENTS

  • Evaluation spans PINWHEEL, Japanese earthquakes, New Jersey COVID-19, BOLD5000 fMRI, and NYC bike-share events, with temporal and spatial baselines.Source: §5 · pp. 7–8
  • On PINWHEEL, each event shifts spatial probability toward the next clockwise cluster, then densities smoothly return to the even initial distribution during inactivity.Source: §5 · p. 7, Figure 5 · p. 8
  • In earthquake snapshots, Jump CNF concentrates density around tectonic plate boundaries, while KDE uses large variance to capture propagation in multiple directions.Source: Figure 7 · p. 8
  • Across datasets, Jump and Attentive CNFs outperform Time-varying CNF, while Time-varying CNF outperforms conditional KDE, suggesting prediction benefits from event-history dependence.Source: §5 · p. 9
  • For COVID-19, Neural STPPs achieve substantially better spatial likelihoods than Hawkes; temporal likelihoods significantly differ from Neural Jump SDEs on every dataset despite often being close.Source: §5 · p. 9
  • Attentive CNF generally improves temporal likelihood while retaining competitive spatial likelihoods, and computes substantially faster than Jump CNF.The authors attribute the likelihood pattern to attention over all prior events versus compressing history into the hidden state.Source: §5 · p. 9

6 CONCLUSION

The paper proposes Neural ODE-based spatio-temporal point-process models combining Neural Jump SDEs and Continuous Normalizing Flows. It reports state-of-the-art performance across varied domains and points to manifold modeling as a future direction.

6 CONCLUSION

  • The proposed parameterizations combine Neural Jump SDEs and Continuous Normalizing Flows to model continuous-space and time events with expressive continuous marks.Source: §6 · p. 9
  • Neural ODEs enable joint-model likelihood computation up to negligible numerical error, and the approach achieves state-of-the-art performance on spatio-temporal datasets across domains.Source: §6 · p. 9
  • Future work could apply the models in earth and climate science, using Riemannian CNFs to model events on manifolds such as Earth’s surface.Source: §6 · p. 9

A ADDITIONAL RESULTS AND FIGURES

The additional figures cover runtime, event-history-dependent spatial distributions, attention weights, and held-out urban-mobility log-likelihoods.

A ADDITIONAL RESULTS AND FIGURES

  • Table 2 reports log-likelihood values on held-out test data for an urban-mobility data set.The supplied caption does not give the values or identify a best-performing model.Source: Table 2 · p. 14
  • Figure 8 compares the runtime of Jump CNF and Attentive CNF.The supplied caption does not state which is faster.Source: Figure 8 · p. 14
  • Figure 9 illustrates Jump CNF and Attentive CNF spatial distributions shifting after events, with each event increasing likelihood for the cluster to its right.The illustration uses synthetic one-dimensional data formed from three Gaussian mixtures.Source: Figure 9 · p. 14
  • Figure 10 presents learned attention weights for random event sequences.Source: Figure 10 · p. 14

B BASELINE

The self-excitation baseline models event times with a Hawkes process and locations with a history-conditioned Gaussian mixture, while its simpler Gaussian spatial model assumes local, direction-independent excitation.

B BASELINE

  • The self-excitation baseline combines a Hawkes temporal model with a Gaussian-mixture spatial distribution conditioned on event history.Its likelihood decomposition makes event time independent of previous spatial variables, while the spatial distribution conditions on past events and the current time.Source: B BASELINE · p. 15
  • The simpler Gaussian conditional model assumes new events are likely to appear near previous events.Its two learnable parameters, σ2 and τ, control spatial and temporal decay, respectively.Source: B BASELINE · p. 15
  • The Gaussian spatial baseline assumes propagation is equal in all directions and captures only local self-excitation.The text notes these assumptions can be unsuitable for settings such as earthquakes, epidemics, and taxi demand.Source: B BASELINE · p. 15

C PRE-PROCESSING STEPS FOR EACH DATA SET

The preprocessing describes synthetic pinwheel, earthquake, COVID-19, Citibike, and BOLD5000 sequences, with varying event counts and dependencies across data sets.

C PRE-PROCESSING STEPS FOR EACH DATA SET

  • PINWHEEL data map a 10-dimensional multivariate Hawkes process to continuous locations by assigning dimensions to pinwheel clusters.Sequences contain 4 to 108 events.Source: C PRE-PROCESSING STEPS FOR EACH DATA SET · p. 15
  • Earthquake sequences use Japan events from 1990–2020 with magnitude at least 2.5, split into 30-day windows separated by seven days.After removing the unusually long November 2010–December 2011 sequences, the data contain 950 training, 50 validation, and 50 test sequences; each has 18 to 543 events.Source: C PRE-PROCESSING STEPS FOR EACH DATA SET · p. 15
  • COVID-19 sequences use New Jersey county-level daily cases from March–July 2020, dequantized across county space and within each day.Seven-day sequences are sampled every three days from March 15; each event is retained with probability 0.01, with 50 samples per sequence, yielding 1450 training, 100 validation, and 100 test sequences.Source: C PRE-PROCESSING STEPS FOR EACH DATA SET · p. 15
  • CITIBIKE treats New York City trip starts from April–August 2019 as events and splits them into one-day sequences starting at 5:00 a.m.Each sequence is subsampled at probability 0.005, 20 times, producing 2440 training, 300 validation, and 320 test sequences.Source: C PRE-PROCESSING STEPS FOR EACH DATA SET · p. 15
  • BOLD5000 preprocessing uses one participant’s fMRI sessions, splits each run into three sequences, and converts brain responses to events using z-score threshold γ = 6.0.The split contains 1050 training, 150 validation, and 220 test sequences, with 6 to 1741 events per sequence.Source: C PRE-PROCESSING STEPS FOR EACH DATA SET · p. 16
  • Across the processed data sets, event counts vary substantially and event dependencies differ, making them difficult for traditional point-process models.Figure 11 shows histograms of events per sequence for each data set.Source: C PRE-PROCESSING STEPS FOR EACH DATA SET · p. 16, Figure 11 · p. 16

D HYPERPARAMETERS CHOSEN AND TESTED

The experiments use specified CNF, hidden-state, and attention architectures, with dataset-wide likelihood and solver settings and selected regularization choices.

D HYPERPARAMETERS CHOSEN AND TESTED

  • CNF drift uses an MLP with dimensions [d −64 −64 −64 −d], and the experiments use time-dependent Swish activation.The paper compared softplus with time-dependent Swish and chose the latter for all experiments; all Neural ODEs were initialized with zero drift.Source: D HYPERPARAMETERS CHOSEN AND TESTED · pp. 16–17
  • The hidden-state drift MLP was mostly set to widths 32 −32, while GRU updates replaced an unstable MLP parameterization for instantaneous changes on long sequences.The tested hidden widths included [8 −20], [32 −32], [64 −64], [32 −32 −32], and [64 −64 −64].Source: D HYPERPARAMETERS CHOSEN AND TESTED · p. 16
  • Hidden-state drift was L2-regularized at strength 1e-4; Jump CNF benefited from regularization to avoid numerical instability.Time-varying and Attentive CNF models mostly used no regularization; optional gradient regularization was also tested for CNFs.Source: D HYPERPARAMETERS CHOSEN AND TESTED · p. 16
  • CNF models use the time interval [2, T + 2] to parameterize a non-trivial spatial distribution over the data interval.The time variable is described as a dummy variable, so the chosen interval does not limit the model.Source: D HYPERPARAMETERS CHOSEN AND TESTED · p. 17
  • Jump CNF instantaneous updates use four composed radial flows whose parameters are produced from the pre-update hidden state.The radial flows were initialized with log determinant near zero.Source: D HYPERPARAMETERS CHOSEN AND TESTED · p. 17
  • Attentive CNF uses Lipschitz multihead attention in a residual branch with ActNorm instead of LayerNorm.Lipschitz attention was generally as stable in validation NLL as standard attention and was more stable on multiple occasions; an auxiliary non-attentive CNF maps from time 0 to the data interval start.Source: D HYPERPARAMETERS CHOSEN AND TESTED · p. 17
  • Reported likelihoods are per-event nats after spatial standardization, and integrals use relative and absolute tolerances of 1E-4 or 1E-6.The tolerances were selected based on preliminary convergence and stability testing.Source: D HYPERPARAMETERS CHOSEN AND TESTED · p. 17
  • Neural Jump SDE uses five Gaussian mixtures for its spatial model, with a hidden-state MLP predicting mixture means, log standard deviations, and coefficients.Source: D HYPERPARAMETERS CHOSEN AND TESTED · p. 17

E REMOVING CROSS-EVENT PARTIAL DERIVATIVES

The section reduces gradient computation by removing cross-event dependencies while retaining causal attention over prior events.

F PARALLEL SOLVING OF MULTIPLE ODES WITH VARYING INTERVALS

The section describes solving ODE systems with different time intervals by reparameterizing them onto a shared unit interval, then integrating them together. It also reports comparisons among history-conditioned models and notes that coupled systems can still be solved as one ODE.

Loading 2011.04583v3…