Source-linked AI summary
Modeling The Intensity Function Of Point Process Via Recurrent Neural Networks
Shuai Xiao, Junchi Yan, Stephen M. Chu, Xiaokang Yang, Hongyuan Zha
TL;DR
Point-process intensity modeling must accommodate arbitrary event distributions while representing both time-varying background behavior and history effects. The paper fuses a time-series RNN with an event-sequence RNN in an end-to-end nonlinear intensity model. On ATM predictive-maintenance data, the fused intensity RNN outperforms its single-RNN counterparts for event-type and timestamp prediction, with especially strong results on subtype prediction.
Problem
Parametric point-process intensity forms have restricted expression capability for arbitrary event data, motivating a flexible nonlinear mapping without prior knowledge of the dynamics.
Method
The model fuses a time-series RNN for spontaneous background dynamics with an event-aligned RNN for long-range history effects, trained end-to-end.
Results
The fused intensity RNN outperforms time-series and event-sequence RNN counterparts for both event-type and timestamp prediction in the studied dataset.
Takeaways & Limitations
The results support using fused RNNs as a flexible, end-to-end approach for point-process modeling in the studied predictive-maintenance application.
Abstract
from arXiv · showhide
Event sequence, asynchronously generated with random timestamp, is ubiquitous among applications. The precise and arbitrary timestamp can carry important clues about the underlying dynamics, and has lent the event data fundamentally different from the time-series whereby series is indexed with fixed and equal time interval. One expressive mathematical tool for modeling event is point process. The intensity functions of many point processes involve two components: the background and the effect by the history. Due to its inherent spontaneousness, the background can be treated as a time series while the other need to handle the history events. In this paper, we model the background by a Recurrent Neural Network (RNN) with its units aligned with time series indexes while the history effect is modeled by another RNN whose units are aligned with asynchronous events to capture the long-range dynamics. The whole model with event type and timestamp prediction output layers can be trained end-to-end. Our approach takes an RNN perspective to point process, and models its background and history effect. For utility, our method allows a black-box treatment for modeling the intensity which is often a pre-defined parametric form in point processes. Meanwhile end-to-end training opens the venue for reusing existing rich techniques in deep network for point process modeling. We apply our model to the predictive maintenance problem using a log dataset by more than 1000 ATMs from a global bank headquartered in North America.
Introduction
The paper treats point-process intensity as a combination of spontaneous background dynamics and history-event effects, then models these components with complementary RNNs. The resulting end-to-end model targets flexible intensity modeling and predictive maintenance on ATM event logs.
- Motivation: Event timestamps provide information about system dynamics beyond the fixed indexing of regularly sampled time series.The paper motivates point processes as a framework for modeling asynchronously generated events and their timestamps.
- Problem formulation: The paper views conditional intensity as a flexible nonlinear mapping from participant, event-profile, and system-history information to transient event occurrence intensity.This avoids restricting the intensity to a specialized parametric form.
- Problem formulation: Many conditional intensity functions combine a spontaneous background component with effects from historical events.The background may reflect time-varying attributes and event type, while history events capture additional effects.
- Approach: A time-series RNN tracks spontaneous background dynamics, while an event-aligned RNN captures long-range dependencies across asynchronous history events.The two RNNs use units aligned respectively with regular time points and event occurrences.
- Approach: The fused model is simple, general, and end-to-end trainable, with output layers for event type and timestamp prediction.The study applies it to decades of event logs from more than 1,000 ATMs at a North American global bank.
Related Work and Motivation
The related work reviews RNNs for synchronized and asynchronous sequences and point-process intensity formulations. Predictive maintenance motivates the application because equipment-risk prediction can support proactive maintenance and resource use.
- Recurrent neural networks: RNNs are used for both evenly spaced series and asynchronous event sequences with random timestamps.For asynchronous events, elapsed time between events is included alongside other features as RNN input.
- Point processes: A point process uses its conditional intensity function to represent the rate of a new event conditioned on the observed history.The conditional intensity is central to point-process modeling, and different processes vary in how it is parameterized.
- Point processes: Poisson, reinforced Poisson, Hawkes, reactive, and self-correcting processes encode different assumptions about background or history effects.Examples include history independence, rich-get-richer behavior, excitation, inhibition, and event-driven decrease of the background.
- Motivation: Parametric intensity forms can have restricted expression capability for arbitrarily distributed event data and may require specialized models based on prior knowledge.The paper therefore separates spontaneous background and history-event effects as a general modeling perspective.
- Predictive maintenance: Predictive maintenance uses equipment-risk prediction to support proactive corrective scheduling and more cost-effective deployment of limited resources.The application uses event data from more than 1,000 ATMs at a global bank headquartered in North America.
Network Structure and End-to-End Learning
The model combines two RNNs: one processes evenly spaced time-series inputs for background intensity, while the other processes asynchronous event sequences for long-range dependencies. Their outputs support joint event-type and timestamp prediction through an end-to-end loss.
- Network Structure: The model uses LSTM units to learn long-range dependencies from recurrent inputs and states.The paper implements its RNNs with LSTMs, while noting that GRUs could also be used.
- Network Structure: Two RNNs separately model evenly spaced time-series inputs and randomly timed event sequences.The time-series RNN models background intensity, while the event-sequence RNN captures long-range event dependency.
- End-to-End Learning: The total loss sums timestamp-prediction loss with cross-entropy loss for event-type prediction.Timestamp predictions also feed the classification loss layer to penalize disagreement with the actual timestamp.
- End-to-End Learning: A Gaussian penalty with fixed σ^2 = 10 encourages predicted timestamps to remain close to ground truth.The objective reinforces timestamp accuracy in addition to correct event-type classification.
- End-to-End Learning: Inverse sample-frequency weights emphasize event classes with fewer training examples during type-prediction training.The weighting is applied to main-type and subtype outputs, and can be disabled selectively for independent prediction losses.
Experiments on Real-world Data
The study evaluates the proposed end-to-end RNN point-process model on ATM maintenance logs, predicting event types and timestamps from synchronized time-series and asynchronous event-sequence information. Results compare architectural variants and established baselines across event-type, subtype, timestamp, and joint metrics.
- Data and task: The dataset contains maintenance logs from 1557 ATMs across five machine models, with error and ticket events recorded between September 2014 and March 2015.Training uses 1085 ATMs and testing uses 469 ATMs.
- Features and model: The fused network combines two LSTMs through an embedding layer and predicts main event type, subtype, and timestamp with separate output layers.Training is end-to-end, using cross-entropy with a time penalty for event types and squared loss for timestamps.
- Features and model: The time-series RNN uses seven-day sub-windows with inventory and event-statistics features, while the event-sequence RNN uses event types and inter-event intervals.The two RNN inputs represent synchronized system information and asynchronous event history.
- Evaluation: Evaluation uses Precision, Recall, F1, confusion matrices, and timestamp MAE, plus stricter joint metrics that condition type or timestamp evaluation on the other prediction being accurate.F1+ is computed when timestamp MAE is below three days, while MAE+ is computed only for correctly predicted event types.
- Architectural comparison: Flat prediction outperforms hierarchical prediction for main types, whereas the hierarchical layer performs better for subtypes by incorporating main-type and embedding representations.The comparison covers different RNN inputs and evaluation metrics.
- Results: The fused intensity RNN outperforms the time-series and event-sequence RNN variants for both event type and timestamp prediction, while generally outperforming peer methods on challenging tasks.Logistic classification performs better on main-type prediction, but the proposed method is especially strong for subtype prediction; point-process methods also outperform classical classification there.
Conclusion
The paper positions its model as a fully implicit approach to point-process intensity modeling, extending beyond explicit and partially parametric formulations. Its simple composite RNN architecture supports end-to-end learning and application without requiring pre-specified dynamics.
- Conclusion: The model advances point-process intensity modeling from explicit and partially parametric formulations toward a fully implicit mapping.Hawkes uses a full explicit parametric model, while RMTPP is partially parametric and omits dense time-series features.
- Conclusion: Figure 4 compares confusion matrices for main-type and subtype predictions across hierarchical and flat structures.The figure presents three methods in the top and middle rows using the hierarchical and flat structures, respectively.
- Conclusion: Figure 5 summarizes the evolution of point-process modeling.It is used to position the proposed model relative to explicit and implicit intensity-function approaches.
- Conclusion: The composite RNN architecture can be learned end-to-end with standard backpropagation.This creates opportunities to reuse neural-network learning advances for point-process modeling and applications.