Source-linked AI summary
Transformer Hawkes Process
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, Hongyuan Zha
TL;DR
Existing point-process models struggle with complex short- and long-term event dependencies, especially because recurrent architectures have limited long-range modeling and trainability. The paper proposes Transformer Hawkes Process, adapting self-attention to continuous-time event sequences. Across real datasets, THP achieves strong likelihood and event-prediction performance, extends to relational multiple point processes, and remains computationally efficient.
Problem
Existing recurrent neural-network point-process models struggle to capture long-term dependencies in complex event sequences.
Method
THP adapts self-attention to continuous-time event sequences, with temporal encoding and extensions for relational multiple point processes.
Results
Experiments on various real-world datasets show state-of-the-art likelihood and event-prediction accuracy, with improved performance when relational information is incorporated.
Takeaways & Limitations
THP provides a general framework for efficient event-stream analysis that can incorporate structural knowledge such as graph relationships.
Takeaways & Limitations
Neural Hawkes models inherit recurrent-network weaknesses, including difficulty capturing long-term dependencies and training difficulty.
Abstract
from arXiv · showhide
Modern data acquisition routinely produce massive amounts of event sequence data in various domains, such as social media, healthcare, and financial markets. These data often exhibit complicated short-term and long-term temporal dependencies. However, most of the existing recurrent neural network based point process models fail to capture such dependencies, and yield unreliable prediction performance. To address this issue, we propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies and meanwhile enjoys computational efficiency. Numerical experiments on various datasets show that THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. Moreover, THP is quite general and can incorporate additional structural knowledge. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.
1 Introduction
Event sequences contain complex temporal dependencies, but recurrent and convolution-based point-process models have important limitations. THP uses self-attention to model dependencies adaptively while retaining computational efficiency and extending to relational event data.
- Event sequence data arise across social media, financial transactions, and personalized healthcare.
- Classical point-process assumptions can oversimplify event relationships, such as treating all past events as positively influential.
- RNN-based models struggle with both long-term dependencies and training scalability because recurrence limits depth and parallel processing.
- Convolution-based sequence models can impose unnecessary dependencies while increasing computational burdens.
- THP uses self-attention to adaptively select dependencies at any temporal distance, capturing short- and long-term effects.Attention scores represent dependency strength between events.
- Experiments on five datasets report superior likelihood and event-prediction performance to RNN-based models, while structured-THP improves learning of relational multiple point processes.
2 Background
Traditional Hawkes and neural Hawkes processes provide continuous-time event modeling but simplify event influences or inherit recurrent-network weaknesses. Transformer architectures offer an alternative, though they require adaptation to irregular event timing.
- Point processes model discrete events occurring in continuous time and include Hawkes and Poisson processes.
- The classical Hawkes formulation assumes every past event positively contributes to current-event occurrence, with influence decaying over time.
- Neural Hawkes Process parameterizes event intensity with recurrent neural networks and uses continuous-time LSTM hidden states for prediction.
- Neural Hawkes inherits recurrent-network limitations, including difficulty capturing long-term dependencies and training difficulty.
- Transformers are attention-based models, but their standard architecture is not directly applicable to point processes because event intervals can be arbitrary.
3 Model
THP encodes event types and timestamps, then uses masked multi-head self-attention and feed-forward layers to represent event histories. A continuous-time conditional intensity supports prediction and likelihood-based training.
- 3.1 Transformer Hawkes Process: Self-attention assigns dependency scores directly among events, allowing THP to select history events at any temporal distance and capture short- and long-term dependencies.Unlike sequential RNN representations, attention can directly model dependencies between the current event and distant history.
- 3.1 Transformer Hawkes Process: Temporal encoding deterministically represents each event timestamp, while an embedding matrix represents event types in the same M-dimensional space.The sequence embedding combines the temporal encodings and event-type embeddings into X, with one row per event.
- 3.1 Transformer Hawkes Process: Masked attention prevents each event from attending to future positions, preserving the temporal direction of event-sequence modeling.Future positions are masked before softmax so they receive no dependency weight.
- 3.1 Transformer Hawkes Process: Stacking self-attention modules and applying a position-wise feed-forward network produces hidden representations that encode increasingly high-level event dependencies.The resulting matrix H contains one hidden representation for each input event.
- 3.2 Continuous Time Conditional Intensity: THP defines type-specific continuous-time intensities from interpolated current influence, hidden-state history, and base intensity, using softplus to keep intensities positive and computation stable.The hidden states encode past events, while the base intensity represents event occurrence without history information.
- 3.3 Training: Next-event time and type predictions are derived from the conditional intensity, and model parameters are learned by maximizing sequence log-likelihood.Because softplus prevents closed-form evaluation of the non-event integral, Monte Carlo integration is used as one approximation.
4 Structured Transformer Hawkes Process
Structured THP incorporates graph-based relational information into a shared model for multiple point processes. Vertex embeddings and learned similarity-based graph attention provide a flexible representation that can extend to complex structured data.
- Structured representation: A graph G=(V,E) represents point-process relationships, with vertices denoting processes and edges denoting relational information.The graph also indicates potential interactions among vertices.
- Structured representation: THP models multiple point processes with a shared architecture while using vertex embeddings to handle heterogeneous processes.Each graph vertex is associated with a point process, and vertex embeddings represent process-specific differences.
- Vertex embeddings: Structured THP augments event and temporal embeddings with vertex embeddings for each event’s associated process.The vertex embedding matrix supplies an M-dimensional embedding for each vertex.
- Graph attention: Graph self-attention uses a learned vertex similarity matrix A, whose entries quantify similarity between pairs of vertices.The attention output uses Q, K, and Vvalue together with A and a learned metric Ω; multiple heads use different metric matrices.
- Graph attention: Stacked multi-head self-attention learns higher-level and more complicated similarities among vertices, including on dynamically evolving graphs.This stacking distinguishes structured THP from RNN-based shallow models.
- Graph regularization: Graph regularization encourages attention between connected vertices while allowing learned similarities to differ from the adjacency matrix.The graph acts as a flexible guide rather than imposing identical behavior on all connected vertices.
5 Experiments
Experiments across diverse event-sequence datasets show that THP improves likelihood, event prediction, interpretability, and efficiency over recurrent baselines. Structured variants further exploit relational information, with THP-S addressing weaknesses of unrestricted attention.
- 5.3–5.4 Likelihood and Event Prediction: THP outperforms RNN-based models in validation likelihood and event prediction across five datasets.The evaluation uses per-event log-likelihood and held-out event prediction accuracy; datasets span social media, finance, healthcare, and online communities.
- 5.3 Likelihood Comparison: Attention heads learn different dependency patterns: early layers focus on individual events, whereas final-layer patterns are more uniformly distributed.The authors attribute the deeper-layer pattern to feature transformations produced by shallower attention heads.
- 5.4 Event Prediction Comparison: THP maintains strong performance across sequence lengths ranging from an average of 4 events in MIMIC-II to 2074 in Financial Transactions.The results are presented as evidence that THP captures both short-term and long-term dependencies.
- 5.5 THP vs. Structured-THP: Relational information improves likelihood, while THP-S further improves event prediction by restricting attention based on spatial closeness of vertices.Full attention can behave inconsistently with few training events or inhomogeneous data; THP-S removes undesirable attention dependencies.
- 5.6 Ablation Study: THP achieves strong performance with approximately 100–200k parameters, compared with about 1000k for NHP and 2000k for TSES.The authors conclude that THP is efficient in both model size and training speed.
6 Conclusion
The paper presents Transformer Hawkes Process (THP), which uses self-attention to model short- and long-term dependencies in event streams efficiently. Experiments show state-of-the-art likelihood and event prediction accuracy, while the framework can incorporate structural knowledge.
- THP uses self-attention to capture sophisticated short-term and long-term dependencies in event streams.
- THP is computationally efficient while analyzing event sequence data.
- Experiments on various real-world datasets show state-of-the-art performance in likelihood and event prediction accuracy.
- THP can integrate structural knowledge for analyzing more complicated data, including event sequences on graphs.