Source-linked AI summary
Fully Neural Network based Model for General Temporal Point Processes
Takahiro Omi, Naonori Ueda, Kazuyuki Aihara
TL;DR
The paper addresses the restrictive intensity-function forms commonly used in RNN-based temporal point-process models. It models the integrated intensity with a feedforward neural network and differentiates it to obtain intensity, enabling exact likelihood evaluation. The model achieves competitive or superior performance on synthetic and real datasets.
Problem
RNN-based point-process models often assume specific intensity-function time courses, which can restrict their expressive ability and predictive skill.
Method
The model represents the cumulative hazard function with a constrained feedforward neural network and obtains the hazard function by differentiation.
Results
The model achieves competitive or superior performance against other models on synthetic and real datasets.
Takeaways & Limitations
The approach provides flexible hazard-function modeling while exactly and efficiently evaluating the log-likelihood without numerical approximation.
Takeaways & Limitations
The study does not consider event marks, focusing instead on developing a general hazard-function model.
Abstract
from arXiv · showhide
A temporal point process is a mathematical model for a time series of discrete events, which covers various applications. Recently, recurrent neural network (RNN) based models have been developed for point processes and have been found effective. RNN based models usually assume a specific functional form for the time course of the intensity function of a point process (e.g., exponentially decreasing or increasing with the time since the most recent event). However, such an assumption can restrict the expressive power of the model. We herein propose a novel RNN based model in which the time course of the intensity function is represented in a general manner. In our approach, we first model the integral of the intensity function using a feedforward neural network and then obtain the intensity function as its derivative. This approach enables us to both obtain a flexible model of the intensity function and exactly evaluate the log-likelihood function, which contains the integral of the intensity function, without any numerical approximations. Our model achieves competitive or superior performances compared to the previous state-of-the-art methods for both synthetic and real datasets.
1 Introduction
Temporal point processes model discrete-event sequences by relating future event occurrences to past history. Existing RNN models improve prediction but often impose restrictive intensity-function dynamics, motivating a more flexible model with exact likelihood evaluation.
- Motivation: Temporal point processes model discrete-event time series and capture dependence between event occurrences through event history.Applications include financial transactions, social-network communication, and website activity.
- Existing models: Conditional intensity functions map past event history to the point-process intensity.This representation characterizes how event history influences future event occurrence.
- Existing models: RNN-based point-process models use hidden states to represent event history and have outperformed parametric models in prediction.Earlier parametric examples include Poisson and Hawkes processes.
- Problem: RNN-based models commonly assume a specific time course for conditional intensity, such as exponential increase or decrease after the latest event.Such assumptions can limit expressive ability and potentially reduce predictive skill when they are inappropriate.
- Proposed approach: The proposed approach models the integral of conditional intensity with a feedforward neural network, then obtains intensity by differentiation for exact log-likelihood evaluation.This avoids numerical approximations while retaining a general intensity model.
2 Method
The method combines an RNN representation of event history with a feedforward network for the cumulative hazard function. Differentiation yields a flexible hazard function, while cumulative-hazard reformulation enables exact likelihood evaluation and efficient median prediction.
- RNN event-history representation: The RNN encodes event history using inputs derived from event times, such as inter-event intervals or their logarithms.The hidden state is treated as a compact representation of the event history and may depend on only the most recent events.
- Problem with prior models: Previous models impose specific hazard-function time courses, such as exponential decrease or increase with elapsed time.Such assumptions can limit expressive ability, while more complex hazard models make likelihood evaluation difficult because it contains an integral.
- Cumulative-hazard network: The proposed model represents the cumulative hazard function Φ(τ|h_i) with a feedforward neural network rather than directly specifying the hazard function.The network is designed to produce positive, monotonically increasing outputs using positive-valued activations and constrained positive weights.
- Hazard construction: The hazard function is obtained as the derivative of the cumulative hazard network output with respect to elapsed time.Automatic differentiation computes this derivative within neural-network libraries.
- Exact likelihood: Rewriting the log-likelihood through the cumulative hazard removes its integral term, allowing exact evaluation for complex cumulative-hazard models.This avoids the numerical approximations used by some flexible intensity models, which can reduce fitting accuracy and increase computational cost.
- Prediction: The cumulative hazard also supports median next-event prediction by solving the cumulative-hazard relation with a root-finding method.The reported implementation generates predictions for 20000 events in only a second.
3 Related works
Previous RNN point-process models commonly impose constant, exponential, or piecewise-constant hazard forms. More flexible models exist, but numerical integration can increase computational cost and affect fitting accuracy.
- Most RNN point-process models assume a specific hazard-function time course, commonly constant or exponential.
- Piecewise-constant hazard models discretize the continuous hazard function into intervals.
- Continuous-time LSTM models represent intensity through continuously evolving outputs.
- Existing general hazard models use numerical approximations to evaluate the integral in the log-likelihood, which can be computationally expensive and affect fitting accuracy.
4 Experiments
The experiments compare four RNN-based point-process models on synthetic and real datasets using held-out predictive performance. The neural network hazard model is competitive or superior across datasets and remains close to the true model on synthetic data.
- Experimental setup: Four RNN-based point-process models are evaluated on synthetic and real data with 64 hidden units, using held-out test sets for prediction.
- Synthetic data: Synthetic benchmarks include stationary and non-stationary Poisson, renewal, self-correcting, and Hawkes processes.
- Synthetic results: The proposed neural network model performs competitively or better than the other models and stays close to the true model across synthetic datasets.
- Synthetic results: Constant and exponential models perform well when their hazard assumptions are correct but poorly under misspecification.
- Synthetic results: The neural network model outperforms the piecewise-constant model particularly on renewal and Hawkes2 processes, where intervals vary substantially and intensity can change rapidly.
- Real data: Real datasets include Nikkei 225 mini transactions, San Francisco emergency calls, MemeTracker phrases, and music-related event sequences.
- Real-data results: On real datasets, the neural network model is competitive or superior overall, with differences greater than 0.5 against other models particularly for Meme and music data.
- Additional comparison: The model is also evaluated against CT-LSTM using mean negative log-likelihood and mean absolute error with nearly equal parameter counts.
5 Discussion and Conclusions
The model represents the hazard function's time course generally with a neural network and offers exact, efficient log-likelihood evaluation. The study also notes that marked events were not considered, although the approach can be extended to marked temporal point processes.
- The proposed approach extends RNN point-process models by representing the hazard-function time course generally with a neural network.
- The model uses a feedforward neural network to reproduce any hazard-function time course in principle.
- Modeling the cumulative hazard instead of the hazard directly avoids numerical integration in the log-likelihood.This enables exact and efficient evaluation without numerical approximation.
- The study does not consider event marks, but states that the approach can be extended to marked temporal point processes.
Evaluating the gradient of the loss function
Training evaluates the negative log-likelihood through a computational graph that uses derivatives of the cumulative hazard. Backpropagation is applied first to obtain the derivative and then to obtain parameter gradients.
- The negative log-likelihood depends on both Zi(τ) and ∂Zi(τ)/∂τ.
- Backpropagation computes ∂Zi(τ)/∂τ before the computational graph evaluates the loss function.
- A second backpropagation pass evaluates the gradient of the loss function with respect to model parameters.The paper calls this double backpropagation.
- Figure S1 depicts the model's network structure for L = 4 while omitting the RNN computational graph for simplicity.