Source-linked AI summary
Learning Temporal Point Processes via Reinforcement Learning
Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, Le Song
TL;DR
Finite-sample limitations of likelihood-based learning motivate an alternative for temporal point processes. The paper treats event generation as a stochastic policy and learns it by comparing generated and observed sequences. Experiments report better performance than MLE across eight datasets and gains over state-of-the-art methods in data description and computational efficiency.
Problem
Finite-sample estimation motivates alternatives to likelihood: MLE lacks finite-sample optimality, while KL-based evaluation can be asymmetric and exhibit high variance or mode dropping.
Source: §7 · p. 9Method
Results
Takeaways & Limitations
Directly minimizing sequence discrepancy is presented as a way to avoid model misspecification and limitations of likelihood-based learning.
Source: §8 · p. 10A more flexible output parameterization via transformed unit-rate exponential samples remains future work and needs a different optimization algorithm.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Social goods, such as healthcare, smart city, and information networks, often produce ordered event data in continuous time. The generative processes of these event data can be very complex, requiring flexible models to capture their dynamics. Temporal point processes offer an elegant framework for modeling event data without discretizing the time. However, the existing maximum-likelihood-estimation (MLE) learning paradigm requires hand-crafting the intensity function beforehand and cannot directly monitor the goodness-of-fit of the estimated model in the process of training. To alleviate the risk of model-misspecification in MLE, we propose to generate samples from the generative model and monitor the quality of the samples in the process of training until the samples and the real data are indistinguishable. We take inspiration from reinforcement learning (RL) and treat the generation of each event as the action taken by a stochastic policy. We parameterize the policy as a flexible recurrent neural network and gradually improve the policy to mimic the observed event distribution. Since the reward function is unknown in this setting, we uncover an analytic and nonparametric form of the reward function using an inverse reinforcement learning formulation. This new RL framework allows us to derive an efficient policy gradient algorithm for learning flexible point process models, and we show that it performs well in both synthetic and real data.
1 Introduction
The paper frames temporal point-process learning as a policy-learning problem, addressing the limits of restrictive intensity assumptions and likelihood-based fitting. It uses inverse reinforcement learning to derive a reward for training a recurrent stochastic policy, and reports strong performance on synthetic and real data.
1 Introduction
- Temporal point processes model continuous-time event data, but parametric intensity assumptions may be too restrictive when the real generative process is unknown.When the assumptions are correct, these models can work well.Source: §1 · p. 1
- More expressive intensity models still rely on likelihood maximization, while a cited GAN approach requires training a discriminator and careful tuning to reach equilibrium.The passage describes this GAN approach as computationally intensive.Source: §1 · p. 1
- An RNN policy is trained to mimic observed event sequences, with generated events fed back into the network and each event associated with a reward.The framework learns the policy by maximizing expected cumulative rewards.Source: Figure 1 · p. 2
- The paper treats each generated event as an action from a stochastic policy, recasting intensity-function learning as policy learning.The policy is parameterized by an RNN over event history, and inverse reinforcement learning is used to learn the unknown reward [15].Source: §1 · p. 2
- The approach outperforms state-of-the-art methods on data description and computational efficiency across experiments on synthetic and real event sequences.The introduction reports these as overall experimental findings.Source: §1 · p. 2
2 Preliminaries
The preliminaries define temporal point processes through their history-conditioned intensity and likelihood, then introduce RKHS embeddings as a way to measure differences between distributions. Likelihood normalization can require costly numerical approximation when the intensity has no simple form.
2 Preliminaries
- A temporal point process is a sequence of discrete event times, and its history-conditioned intensity uniquely characterizes the event-generating process.Different intensity forms capture different event-generating patterns.Source: §2 · p. 2
- The likelihood of an event at time t is its intensity multiplied by the conditional survival probability, while likelihood normalization can require costly numerical approximation.The approximation may affect fitting accuracy when the intensity has no simple form.Source: §2 · pp. 2–3
- An RKHS embeds distributions as elements of a function space, where the distance between embeddings measures distributional discrepancy.For characteristic kernels, zero embedding distance holds if and only if the distributions are equal.Source: §2 · p. 3
3 A Reinforcement Learning Framework
The framework treats observed event trajectories as demonstrations from an expert policy and learns a stochastic policy that generates similar sequences. It connects policy actions to event times and intensity, then uses inverse reinforcement learning to obtain a reward that supports policy improvement.
3 A Reinforcement Learning Framework
- Observed event paths are treated as expert demonstrations, and learning seeks a policy whose generated sequences have similar patterns.The number of events may differ across trajectories.Source: §3 · p. 3
- The policy samples an inter-event time as its action, advances the next event time, and updates the history with that event.The policy corresponds to the conditional probability of the next event time and thereby determines the point-process intensity.Source: §3 · p. 3
- Because the reward is unknown, inverse reinforcement learning seeks a reward that gives the expert higher expected cumulative reward than learner policies.The learner can then approach expert performance by maximizing that reward.Source: §3 · pp. 3–4
- Restricting reward functions to the RKHS unit ball yields an analytical reward update from finite expert trajectories and learner roll-outs, avoiding repeated inner-loop reinforcement-learning solves.The reward quantifies discrepancy between expert and learner policies or intensities; policy gradient updates the learner to close the gap.Source: §3 · p. 4
4 Model
The model uses a recurrent stochastic policy to generate inter-event times from event history, and learns it by minimizing the maximum expected reward discrepancy between expert and learner. Restricting rewards to the unit ball of an RKHS yields an analytical optimal reward and a simpler, empirically evaluable objective.
4 Model
- A recurrent neural network with stochastic neurons samples positive inter-event times from a distribution parameterized by event history.Each sampled time is fed back into the network, combining the previous hidden state with the latest sampled action.Source: §4 · p. 4
- The reward class is the unit ball of an RKHS, balancing flexibility with finite-sample learnability and enabling the simpler minimization formulation.The framework can use other kernels; the experiments use a Gaussian kernel.Source: §4 · p. 5
- Expert and learner trajectories are represented through mean embeddings, whose empirical means can estimate the policy discrepancy from sampled sequences.The method presents finite-sample estimation using trajectories from both policies.Source: §4 · p. 5
- Theorem 1 reduces inverse reinforcement learning to minimizing the maximum expected cumulative reward discrepancy between expert and learner policies.The discrepancy can be empirically evaluated, and the optimal reward has an analytical expression.Source: §4 · p. 5
5 Learning Algorithm
The learning algorithm updates a recurrent stochastic policy by policy gradient using a reward derived from the current expert–learner discrepancy. It uses sampled trajectories to estimate rewards and reduce gradient variance; the paper contrasts this approach with likelihood fitting and adversarial min-max training.
5 Learning Algorithm
- The method learns the policy by minimizing the squared expert–learner discrepancy with a policy-gradient update.The squared discrepancy is used because squaring is a monotonic transformation.Source: §5 · pp. 5–6
- Each iteration samples expert and learner trajectories, using one learner trajectory for evaluation and the others to estimate the reward.The reward estimate also uses the expert trajectories.Source: §5 · p. 6
- A reward-to-go term and baseline reduce variance in the policy-gradient estimate.The gradient construction exploits that future actions do not depend on past rewards.Source: §5 · p. 6
- Unlike likelihood fitting, the approach compares generated events with observations during training and bypasses the likelihood’s log-survival integral.The paper says this can avoid model misspecification while retaining a statistically principled interpretation through the conditional density.Source: §5 · p. 6
- The authors characterize the approach as more stable and efficient than GAN or GAIL min-max training, citing a closed-form discrepancy and an unbiased gradient estimator.They contrast its single policy-parameter optimization with the generator and discriminator parameters and nonconvex min-max problems in GAN or GAIL.Source: §5 · pp. 6–7
6 Experiments
Experiments compare RLPP with state-of-the-art and parametric baselines on synthetic and real event sequences. RLPP shows stronger fitting under model misspecification, with competitive results on real data and favorable comparisons against several alternatives.
6 Experiments
- RLPP is evaluated on synthetic IP, Hawkes, and two IP–Hawkes mixtures, plus four real datasets, against WGANTPP, RMTPP, and parametric baselines.The real datasets cover 911 calls, academic citations, clinical visits, and NYSE transactions.Source: §6 · p. 7
- RLPP consistently beats RMTPP and matches or sometimes exceeds WGANTPP on empirical-intensity comparisons across synthetic and real data.The figures compare intensities estimated from generated sequences with those from observed events.Source: §6 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8
- Under model misspecification, RLPP outperforms three conventional parametric models; on Hawkes data it can match the ground-truth model and it performs decently on real datasets.The authors report that RLPP captures major data trends and nonlinear dependencies without prior knowledge of the true parametric form.Source: §6 · p. 8
- On HP and HP+IP1, RLPP’s QQ-plot slope is close to the diagonal; for a Hawkes target, RLPP and learned SE are indistinguishable under the reported KS-test comparison.The KS-test p-value CDF is compared with the diagonal expected under the uniform null distribution.Source: §6 · p. 8, Figure 6 · p. 9, Figure 7 · p. 9
- On real data, RLPP fits similarly to LGCP and better than nonparametric Hawkes; LGCP takes 127 versus 80 minutes on 5% of the data.The authors note that LGCP discretizes time and is typically costly in computation and storage, while nonparametric Hawkes is difficult to generalize beyond the observation window.Source: §6 · p. 9
7 Discussions
Across eight synthetic and real-world datasets, the proposed method consistently outperforms MLE. The discussion motivates its discrepancy measure and describes a more flexible policy variant as future work.
7 Discussions
- Figure 8 compares empirical intensity functions.Source: Figure 8 · p. 9
- Across eight synthetic and real-world datasets, the proposed method consistently outperforms the MLE-based RMTPP model.RMTPP uses a similar RNN outputting exponential-distribution parameters but fits them with maximum likelihood.Source: §7 · p. 9
- Finite-sample MLE lacks optimality guarantees; the authors motivate RKHS discrepancy with two-sample testing work and report favorable finite-sample and asymptotic properties.They note that KL divergence can have high variance and mode dropping in finite samples.Source: §7 · p. 9
- A future variant could transform unit-rate exponential samples rather than specify an output distribution, potentially increasing flexibility but requiring a different optimization algorithm.The authors leave investigation of this variant for future work.Source: §7 · pp. 9–10
8 Conclusions
The paper concludes that reinforcement learning can be used to learn point-process models by matching generated sequences to observations. Its RNN policy uses an analytically derived reward, and the approach performs well on synthetic and real data.
8 Conclusions
- The framework learns point-process models by updating an RNN policy with stochastic neurons that sequentially sample discrete events.Source: §8 · p. 10
- Minimizing discrepancy between generated and observed sequences avoids model misspecification and limitations of likelihood-based approaches.Source: §8 · p. 10
- Choosing the unit ball in RKHS for the reward-function class yields an analytical optimal reward that encourages samples closer to observations.Source: §8 · p. 10
- The proposed approach performs well on both synthetic and real data.Source: §8 · p. 10