Source-linked AI summary
Intensity-Free Learning of Temporal Point Processes
Oleksandr Shchur, Marin Biloš, Stephan Günnemann
TL;DR
Intensity-based temporal point-process models face trade-offs between flexibility, likelihood efficiency, and tractable sampling or moments. The paper directly models conditional inter-event-time distributions with neural density-estimation methods, including normalizing flows and a simple mixture model. The proposed models achieve state-of-the-art or competitive performance and support applications including prediction, sequence embedding, and missing-data imputation.
Problem
Intensity-based temporal point-process models typically trade off flexibility, likelihood efficiency, and ease of sampling or moment computation.
Method
The paper directly models the conditional distribution of inter-event times using normalizing flows and a simple mixture model.
Results
The proposed models achieve state-of-the-art performance in standard prediction tasks, while the mixture model supports closed-form sampling and moment computation.
Takeaways & Limitations
The models support prediction, conditional generation, sequence embedding, and missing-data imputation in temporal point processes.
Takeaways & Limitations
The flow-based models generally lack closed-form expectations and require iterative root finding for sampling.
Abstract
from arXiv · showhide
Temporal point processes are the dominant paradigm for modeling sequences of events happening at irregular intervals. The standard way of learning in such models is by estimating the conditional intensity function. However, parameterizing the intensity function usually incurs several trade-offs. We show how to overcome the limitations of intensity-based approaches by directly modeling the conditional distribution of inter-event times. We draw on the literature on normalizing flows to design models that are flexible and efficient. We additionally propose a simple mixture model that matches the flexibility of flow-based models, but also permits sampling and computing moments in closed form. The proposed models achieve state-of-the-art performance in standard prediction tasks and are suitable for novel applications, such as learning sequence embeddings and imputing missing data.
1 INTRODUCTION
Temporal point processes model irregularly timed human activity, but intensity-based learning typically requires trade-offs among flexibility, efficiency, and ease of use. This paper instead treats learning as conditional density estimation and develops flow-based and mixture models for temporal point processes.
- Temporal point processes represent hospital visits, purchases, financial transactions, and social-media posts as discrete events at irregular intervals.
- Existing intensity-based methods often fall short on flexibility, closed-form likelihood evaluation, or easy sampling and moment computation.
- The paper reframes temporal point-process learning as conditional density estimation rather than conditional-intensity modeling.
- Normalizing flows provide flexible, theoretically sound temporal point-process models, while a simple mixture model matches state-of-the-art performance and supports closed-form sampling and moment computation.
- Experiments apply the proposed models to prediction, conditional generation, sequence embedding, and training with missing data.
2 BACKGROUND
Temporal point processes describe increasing event times and are commonly learned through conditional intensity functions. Intensity parameterizations trade expressiveness against tractable likelihood computation, motivating direct modeling of inter-event-time distributions.
- A temporal point process is a random process whose realizations are sequences of strictly increasing arrival times.
- Inter-event times are strictly positive, and arrival-time and inter-event-time representations are isomorphic.
- Conditional intensity functions specify dependence on event history and support likelihood-based parameter estimation.
- Choosing an intensity form creates a trade-off: simple functions yield closed-form integrals and tractable likelihoods but limited expressiveness, whereas sophisticated functions require Monte Carlo integration.
- FullyNN models the cumulative conditional intensity for tractable likelihoods but do not define a valid PDF, make sampling expensive, and lack closed-form expectations.
- The paper addresses these issues by directly learning the conditional inter-event-time distribution using neural density-estimation methods.
3 MODELS
The paper models positive inter-event times directly with conditional density estimators, using normalizing flows and log-normal mixtures. These models target flexibility while addressing likelihood, sampling, moment-computation, conditioning, and missing-data use cases.
- 3.1 MODELING p(τ) WITH NORMALIZING FLOWS: Normalizing flows model p(τ) by transforming a simple density, with DSFlow and SOSFlow providing flexible density estimators for positive inter-event times.The inverse log transformation maps τ ∈ R+ to the real line before applying parametric flow layers.
- 3.1 MODELING p(τ) WITH NORMALIZING FLOWS: Flow-based models support efficient maximum-likelihood training but generally lack closed-form moments and require iterative root finding for sampling.Their forward transformations cannot generally be computed analytically because the component functions are not analytically invertible.
- 3.2 MODELING p(τ) WITH MIXTURE DISTRIBUTIONS: The proposed log-normal mixture models one-dimensional positive inter-event times and offers simple sampling and finite, computable moments.Mixture weights, means, and standard deviations parameterize the components; samples can be differentiated using reparameterization, including Gumbel-softmax for component selection.
- 3.2 MODELING p(τ) WITH MIXTURE DISTRIBUTIONS: Conditioning uses a context vector that concatenates history, metadata, and sequence embeddings, from which the distribution parameters are generated.This allows the next-event-time distribution to vary with factors such as user metadata or day of the week.
- 3.4 DISCUSSION: Both flow-based models and the mixture model have universal approximation properties, so the mixture can be as expressive as the flow-based alternatives in principle.The conditional formulation additionally assumes that the RNN can encode all relevant history information into its embedding.
- 3.4 DISCUSSION: The mixture model combines flexibility with easy sampling and well-defined moments, while automatically ensuring that the modeled density integrates to one.The paper presents these properties as advantages over intensity-based deep learning models.
4 RELATED WORK
Prior neural temporal point-process models increase flexibility by modeling conditional intensities with neural networks, but this can compromise closed-form likelihood evaluation. Related approaches also include intensity mixtures and neural density-estimation methods.
- Neural point-process models use neural networks to address poor results from simple process misspecification.
- Flexible neural intensity architectures can lose closed-form likelihood evaluation and require Monte Carlo integration.
- RNN-based intensity models encode event history into vectors that parameterize constant or exponential intensities.
- FullyNN models the cumulative intensity with a neural network, paralleling the transformation view used by normalizing flows.
- Kernel-mixture intensity models capture self-excitation but do not support closed-form expectation computation or sampling.
- Neural density estimation comprises mixture-density networks, which output distribution parameters, and normalizing flows, which transform simple distributions.
5 EXPERIMENTS
Experiments evaluate flow-based and log-normal-mixture models on event-time prediction and additional applications. LogNormMix performs strongly across prediction, conditional information, missing-data, and sequence-embedding settings.
- The study evaluates event-time prediction with and without marks, plus conditional information, missing-data training, and sequence embeddings on real-world event datasets.
- 5.1 EVENT TIME PREDICTION USING HISTORY: LogNormMix, DSFlow, SOSFlow, and FullyNN dominate the simpler Gompertz/RMTPP and LogNormal models in event-time prediction.
- 5.2 MARKED TEMPORAL POINT PROCESSES: LogNormMix shows dominant performance for marked temporal point-process prediction on Reddit and MOOC datasets.
- 5.3 LEARNING WITH ADDITIONAL CONDITIONAL INFORMATION: Additional conditional information improves LogNormMix test loss whether or not the history embedding is used.
- 5.4 MISSING DATA IMPUTATION: Sampling missing inter-event times during training generalizes to the true underlying distribution, whereas ignoring or averaging gaps overfits observed data.
- 5.5 SEQUENCE EMBEDDING: Trainable sequence embeddings enable unsupervised differentiation between sequences generated by different distributions.
- 5.5 SEQUENCE EMBEDDING: Interpolating between self-correcting and renewal sequence embeddings maps to different generated inter-event-time distributions.
6 CONCLUSIONS
The paper reframes temporal point-process learning as conditional density estimation rather than direct intensity modeling. Normalizing flows and mixtures retain intensity-based benefits while improving flexibility and tractability.
- The proposed approach directly models the conditional inter-event-time distribution p*(τ) instead of the conditional intensity λ*(t).
- Log-normal mixtures provide a tractable density model whose CDF is defined from the standard-normal CDF.
- Conditional PDFs and CDFs suffice to recover each model’s conditional and cumulative intensity functions.
- Merged independent processes retain additive intensities, with the merged CDF and PDF obtained from the component distributions.
- Direct PDF/CDF modeling does not sacrifice the benefits of intensity parameterization.
B DISCUSSION OF CONSTANT & EXPONENTIAL INTENSITY MODELS
Constant and exponential intensity parameterizations correspond to exponential and Gompertz inter-event-time distributions. These distributions are limited in flexibility, whereas mixtures and flows can approximate arbitrary densities.
- A constant conditional intensity corresponds to an exponential distribution.
- A log-normal mixture can approximate any density while remaining tractable and easy to sample from.
- Gompertz distributions are unimodal and light-tailed for all parameter choices.
- The exponential intensity model is equivalent to a Gompertz distribution after identifying α = exp(d) and β = w.
- Exponential and Gompertz parameterizations therefore have limited flexibility despite rich RNN history embeddings.
C DISCUSSION OF THE FULLYNN MODEL
The FullyNN model can be interpreted through conditional intensity and normalizing-flow perspectives, but its architecture introduces validity and sampling limitations. Specifically, it may define an improper distribution, allow negative inter-event times, and require numerical sampling.
- Model interpretation: FullyNN models the cumulative conditional intensity Λ∗(τ), from which the intensity and PDF are obtained through differentiation and multiplication.The model can also be viewed as transforming positive inter-event times into an Exponential(1) variable.
- Model interpretation: Both the intensity-based and normalizing-flow interpretations lead to the same conclusion about the model.
- Sampling: Sampling from FullyNN is not exact and requires numerical approximation, such as solving an equation with bisection.
- Shortcomings: The FullyNN PDF does not integrate to 1 because its cumulative intensity remains bounded as τ approaches infinity.A valid PDF would require limτ→∞Λ∗(τ)=∞.
- Shortcomings: FullyNN assigns non-zero probability to negative inter-event times, violating the assumption that inter-event times are strictly positive.The model has Λ∗(0)>0, so it permits τ≤0.
D.1 SHARED ARCHITECTURE
The models share a common recurrent and density-estimation pipeline while differing in how they calculate the conditional inter-event-time distribution. The log-normal mixture additionally supports reparameterized sampling, with a small gradient-bias trade-off from its mixture estimator.
- Shared architecture: All models use the same preprocessing, parameter tuning, and model-selection pipeline, differing only in how p∗(τ) is calculated.This common setup is intended to ensure a fair evaluation.
- Shared architecture: An RNN encodes log-transformed, centered inter-event times and optional marks or conditioning information into a history representation h_i.Additional inputs can include metadata and sequence embeddings.
- Shared architecture: Distribution parameters are generated from the concatenated history and conditioning vectors, with transformations enforcing required constraints.
- Flow decoders: Normalizing-flow decoders evaluate p∗(τ) through a base distribution and inverse transformations, enabling maximum-likelihood training.
- Mixture model: The log-normal mixture uses constrained weights, means, and scales and permits reparameterized sampling useful for missing-data imputation.Straight-Through Gumbel estimation passes differentiable relaxed samples backward while using one-hot samples forward.
- Mixture model: The Straight-Through Gumbel estimator produces slightly biased gradients, whereas unbiased alternatives are more expensive to compute.
E.1 SYNTHETIC DATA
The synthetic evaluation uses several standard point processes, while the real-world evaluation covers diverse event-sequence datasets. These datasets span stationary, history-dependent, marked, and social or behavioral event streams.
- Synthetic data: The synthetic benchmark contains 64 sequences per process, with 1024 events in each sequence.
- Synthetic data: The synthetic processes include Poisson, renewal, self-correcting, and two Hawkes datasets.They represent constant-rate, clustered, regularizing, and self-exciting event patterns.
- Real-world data: The real-world datasets include LastFM, Reddit, Stack Overflow, MOOC, Wikipedia, and Yelp.They represent listening, posting, rewards, course interactions, page edits, and restaurant reviews.
- Real-world data: MOOC contains 97 unique interaction types, including video watching and quiz solving.
- Real-world data: Yelp sequences consist of reviews over time for the 300 most visited restaurants in Toronto.
F.1 EVENT TIME PREDICTION USING HISTORY
The evaluation trains models on inter-event-time likelihood across multiple data splits and compares time prediction, marked prediction, and contextual event-time modeling. LogNormMix and DSFlow achieve the strongest reported results, while LogNormMix captures multimodal Yelp behavior and marks improve time prediction.
- Event-time prediction: On synthetic data, LogNormMix and DSFlow achieve the best results and recover the true distribution.
- Event-time prediction: On Yelp, LogNormMix recovers likely next-check-in modes around 0h, 24h, 48h, and subsequent days better than the compared alternatives.RMTPP cannot learn multimodal distributions, while FullyNN struggles to capture this behavior.
- Learning with marks: Using marks as RNN inputs improves time prediction quality for all models, while mark prediction accuracy remains similar across models.The setup assumes marks are conditionally independent of time given the history.
- Contextual event-time prediction: The Yelp context experiment conditions p∗(τ_i) on whether the previous event occurred on a weekday or weekend and during evening hours.Each of the four timestamp contexts receives a learnable 64-dimensional embedding.