Source-linked AI summary
Neural Temporal Point Processes: A Review
Oleksandr Shchur, Ali Caner Türkmen, Tim Januschowski, Stephan Günnemann
TL;DR
Neural TPP research needs a consolidated account of model design, applications, and unresolved challenges. This review organizes general construction principles and application areas, while identifying experimental and domain-level barriers that shape future work.
Problem
Neural TPPs have developed rapidly across architectures and applications, creating a need to consolidate their design principles and broader research landscape.
Method
The paper surveys neural TPP building blocks, model architectures, applications, challenges, and future research directions.
Results
The review provides a general account of neural TPP construction principles and surveys their applications and current challenges.
Takeaways & Limitations
Future work should develop standardized benchmarks, reference implementations, and neural TPP applications for new tasks and domains.
Takeaways & Limitations
Fair comparison remains difficult because standardized experimental setups and high-quality benchmark datasets are lacking.
Abstract
from arXiv · showhide
Temporal point processes (TPP) are probabilistic generative models for continuous-time event sequences. Neural TPPs combine the fundamental ideas from point process literature with deep learning approaches, thus enabling construction of flexible and efficient models. The topic of neural TPPs has attracted significant attention in the recent years, leading to the development of numerous new architectures and applications for this class of models. In this review paper we aim to consolidate the existing body of knowledge on neural TPPs. Specifically, we focus on important design choices and general principles for defining neural TPP models. Next, we provide an overview of application areas commonly considered in the literature. We conclude this survey with the list of open challenges and important directions for future work in the field of neural TPPs.
1 Introduction
Temporal point processes model continuous-time event sequences, while neural TPPs extend them to capture complex dependencies and support efficient modeling. This survey reviews neural TPP design principles, applications, challenges, and future directions.
- Temporal point processes are probabilistic generative models for variable-length event sequences observed in continuous time.Applications include earthquakes, neural spikes, financial trades, and web activity.
- Classical TPPs capture relatively simple patterns, whereas neural TPPs learn complex dependencies and can be computationally more efficient.
- The survey focuses on general principles and building blocks for constructing neural TPP models rather than describing every approach in full detail.
- It reviews neural TPP models and applications, then discusses current challenges and future research directions.
2 Background and Notation
A temporal point process is a distribution over variable-length event sequences in a time interval. Marked TPPs describe events through arrival times and marks, and conditional intensities provide one characterization of their distributions.
- A temporal point process is a probability distribution over variable-length sequences in a time interval [0, T].
- A marked TPP realization contains ordered arrival times, a random event count, and marks drawn from a mark space.Categorical marks are common, but continuous mark spaces are also possible.
- For marked TPPs with K categorical marks, K conditional intensity functions characterize event distributions given the history.The same formulation covers unmarked TPPs by setting K = 1.
3 Autoregressive Neural TPPs
Autoregressive neural TPPs sequentially predict the next event by encoding past events into a fixed-dimensional history vector and using it to parameterize the next-event distribution. Their design balances expressive distributions, efficient likelihood computation, and tractable sampling, with recurrent and set-aggregation encoders offering different computational trade-offs.
- Architecture: Autoregressive neural TPPs sequentially predict each next event’s time and mark from the encoded event history.The procedure represents events as features, encodes prior features into a history embedding, and parameterizes the conditional distribution of the next event.
- Event representation: Event features combine time information, such as arrival or inter-event times, with mark representations such as embeddings for categorical marks.Trigonometric positional features and direct real-valued marks are alternative choices discussed in the review.
- History encoders: Recurrent encoders compute all N history embeddings in O(N) time but require sequential processing and often truncated backpropagation through time.Their linear computation compares favorably with the general O(N^2) likelihood scaling of Hawkes processes, while truncated backpropagation provides approximate gradients.
- History encoders: Set-aggregation encoders compute each history embedding independently and can use self-attention, potentially capturing long-range dependencies better than recurrent encoders.The review notes that this claimed advantage still needs more thorough evaluation and highlights their parallel computation.
- Time prediction: Autoregressive models parameterize the next inter-event-time distribution from the history embedding, for example through θ_i = σ(W h_i + b).The nonlinear function can enforce parameter constraints such as non-negativity, while the density must respect strictly positive inter-event times.
- Distribution design: Choosing a time-distribution parameterization requires balancing flexibility, closed-form likelihood computation, and closed-form sampling.Simple unimodal distributions are tractable but less flexible; invertible splines can provide both flexibility and analytic sampling and likelihood computation, whereas numerical integration is slower and less accurate.
4 Continuous-time State Evolution
Continuous-time neural TPPs maintain a hidden state h(t) throughout the interval, evolving it between events and updating it instantaneously at event times. The state directly determines mark-specific conditional intensities, supporting irregularly sampled and spatio-temporal data but requiring costly numerical approximation.
- State evolution: Between events, Evolve continuously advances the state, while Update applies an instantaneous change when an event occurs.State evolution can use exponential decay or an ordinary differential equation.
- State representation: The state h(t) is defined for every time t and directly determines the conditional intensity for each mark.A nonlinear function g_k maps h(t) to the intensity λ*_k(t).
- Advantages: Continuous-time models naturally handle irregularly sampled time series and support estimating unobserved attributes at arbitrary times.They are also suited to spatio-temporal point processes with continuous-valued marks.
- Trade-offs: Training and sampling are slower because state evolution and likelihood evaluation require numerical approximation of intractable integrals.This is the principal computational cost relative to autoregressive models.
5 Parameter Estimation
Neural TPPs are commonly trained by minimizing negative log-likelihood, although sampling-based objectives provide alternatives for application-specific goals. Computational difficulties arise from likelihood integrals, large mark sets, nondifferentiable event counts, and gradient estimation.
- Maximum likelihood: Negative log-likelihood is the default training objective for both neural and classical TPPs.It combines event contributions with the probability of observing no additional events over the observation interval.
- Maximum likelihood: Likelihood integrals can be approximated by Monte Carlo integration or numerical quadrature, while some autoregressive models compute them analytically.Analytic computation is described as more accurate and computationally efficient.
- Maximum likelihood: When the number of marks K is extremely large, noise-contrastive-estimation approximations can reduce the cost of evaluating the NLL.The exact NLL remains computationally expensive in this regime.
- Alternatives to MLE: Sampling-based objectives train models to make generated sequences similar to training sequences using scoring functions such as Wasserstein, adversarial, or inverse-reinforcement-learning losses.Application-specific examples include rewards, missing-event likelihoods, and ELBO objectives.
- Alternatives to MLE: Sampling-based losses usually require Monte Carlo gradient estimators, and discrete event counts can make them nondifferentiable.Pathwise gradients and differentiable relaxations address these issues when applicable.
6 Applications
Neural TPP applications primarily address prediction and structure discovery in event sequences, with especially strong representation in web-related domains. The review also describes uses in recommendation, mobility, healthcare, social networks, knowledge graphs, and other scientific settings.
- Prediction: Prediction tasks estimate the times and marks of future events from the history of past events.Nearly all reviewed papers include numerical experiments on prediction tasks.
- Prediction: Recommendation models predict the time and identity of the next purchase and can use temporal behavior to time promotions.Neural networks also provide embeddings for large item sets.
- Prediction: Other prediction applications include mobility trajectories, clinical events, assisted-living activity, and demand forecasting in sparse time series.Mobility examples include taxi trajectories and location-based check-ins.
- Structure discovery: Structure discovery tasks infer dependencies among event types, including influence networks and Granger-causality relationships.Neural methods can use integrated gradients to estimate dependencies between event types.
- Other applications: Neural TPPs model information diffusion, evolving social networks, dynamic knowledge graphs, item popularity, speech, and zebrafish behavior.These applications extend beyond the dominant web-related prediction and structure-discovery settings.
7 Open Challenges
The review identifies weaknesses in evaluation practice, benchmark data, metrics, and application coverage as major challenges for neural TPP research. It recommends standardized comparisons, better datasets and libraries, probabilistic evaluation, and broader tasks and domains.
- Experimental setup: Lack of standardized experimental setups and high-quality benchmark datasets makes fair comparison of neural TPP architectures problematic.The authors identify reference implementations and large benchmark datasets as critical next steps.
- Experimental setup: Because new architectures often alter multiple components simultaneously, ablation studies are needed to identify which design choices produce empirical gains.Baseline selection also varies substantially across papers and model families.
- Datasets: Common datasets may not reveal real-world improvements: MIMIC-II and StackOverflow can be solved by a simple history-independent baseline.The usual i.i.d. assumption for training sequences may also be inappropriate for existing datasets.
- Evaluation metrics: NLL obscures separate time and mark prediction quality, while point-estimate metrics are poorly matched to probabilistic trajectory forecasts.The review notes that NLL can be disproportionately affected by mark errors and offers little domain-expert insight.
- Evaluation metrics: Neural TPPs can sample complete future trajectories, representing uncertainty and answering queries beyond single-event point estimates.Examples include querying how many events of one type follow an event of another type.
- Applications: Most neural TPP work focuses on web-related domains, leaving traditional areas such as neuroscience, seismology, and finance comparatively understudied.These domains impose distinct requirements including many marks, interpretability, and complex asset dependencies.
- Applications: Considering new tasks and application domains is identified as an important direction for future neural TPP research.Classical TPPs already support tasks such as clustering, change-point detection, anomaly detection, and optimal control.