Source-linked AI summary
Transformer Embeddings of Irregularly Spaced Events and Their Participants
Chenghao Yang, Hongyuan Mei, Jason Eisner
TL;DR
Irregularly spaced event sequences and event-dependent symbolic databases call for generative models that represent event timing and database facts. This paper replaces recurrent architectures in neural Hawkes and neural Datalog-through-time models with attention-based versions, reporting competitive or better accuracy. Its results support attention-based modeling of continuous-time sequences, while the approach can require expensive embeddings of possible events and, for selective attention, expert-supplied influence structure.
Problem
Method
Results
The models outperform other Transformer-based models on multiple real-world datasets and beat or tie them on multiple synthetic datasets.
Source: §8 · p. 10Takeaways & Limitations
Attention-based models can effectively represent irregularly spaced continuous-time sequences, including in lower-data regimes.
Source: §1 · p. 1
Abstract
from arXiv · showhide
The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-specific pattern-matching rules); future events are then conditioned on the database contents. They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and the possible event has a time-varying embedding that is derived from its symbolic provenance. In this paper, we modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).
1 INTRODUCTION
The paper replaces recurrent architectures for irregularly spaced event sequences with attention-based models, including versions for symbolic database settings. The resulting models remain competitive while enabling broader history representations and parallel computation.
1 INTRODUCTION
- Event embeddings may also serve downstream applied tasks, as word-token embeddings do in BERT.The paper notes that they could be pretrained with a BERT-like objective and fine-tuned to predict properties of observed events.Source: §1 · p. 1
- Events are embedded by attending to previous events with continuous-time positional encodings, and possible future events are embedded at candidate times to predict their instantaneous probabilities.Time-specific queries let attention weights, embeddings, and probabilities vary continuously without explicit decay functions.Source: §1 · pp. 1–2
- The Transformer represents the past with embeddings of every earlier event rather than an LSTM’s fixed-dimensional state, supporting long-distance influences.Its shallower computation graph permits layer embeddings to be computed in parallel during training, though it cannot represent inherently deep concepts such as parity.Source: §1 · p. 1
- The paper progresses from event-context embeddings to generative prediction, then adds expert-written selective-attention rules and Datalog-through-time rules for symbolic facts and possible events.In the final framework, facts and possible events receive time-varying embeddings derived from parameters associated with the rules that established them.Source: §1 · p. 2
- The attention-based NHP and NDTT models achieve comparable or better accuracy, and attention models are effective for irregularly spaced sequences even in lower-data regimes.The conclusion also reports outperforming other Transformer-based models on multiple real-world datasets and beating or tying them on multiple synthetic datasets.Source: §1 · pp. 1–2
2 CONTINUOUS-TIME TRANSFORMER FOR EMBEDDING EVENTS
The section defines event embeddings through attention over a relevant history, using event-type representations and continuous-time positional encodings. Layer-wise computation depends on the preceding layer, allowing embeddings to be computed in parallel.
2 CONTINUOUS-TIME TRANSFORMER FOR EMBEDDING EVENTS
- An event’s embedding is formed by concatenating its layer-wise representations, with each higher layer computed by attending to relevant events in its history.The history may be task-specific; the paper gives a BERT-like pretraining setup using a corrupted sequence of observed events as one example.Source: §2 · pp. 2–3
- Each layer’s attention uses value, key, and query vectors extracted from the preceding-layer event embeddings using learned, layer-specific matrices.The displayed formulas also include the time embedding in those projections.Source: §2 · p. 3
- The base representation is a learned embedding of the event type, while a fixed continuous-time positional encoding represents t because absolute embeddings cannot be learned for all real numbers.The encoding uses sinusoidal coordinates, with cosine and sine forms for alternating dimensions.Source: §2 · p. 3
- Because each layer needs only preceding-layer embeddings from relevant events, all event embeddings at a given layer can be computed in parallel during training.The paper reports that its simplified Transformer equations performed equally well in pilot experiments.Source: §2 · p. 3
3 GENERATIVE MODELING OF CONTINUOUS-TIME EVENT SEQUENCES
The model predicts future event types and times by assigning time-varying intensities to possible events, conditioned on past events. It computes each possible event’s embedding from that history and uses it to determine the event’s intensity.
3 GENERATIVE MODELING OF CONTINUOUS-TIME EVENT SEQUENCES
- At any time t, an event of type e has infinitesimal occurrence probability λ_e(t)dt, where λ_e(t) is its intensity.The probability of an event in [t, t + ϵ) approaches λ_e(t)ϵ as ϵ approaches zero.Source: §3 · p. 3
- The model computes an intensity for each possible event from its top-layer embedding, which is built using only events that occurred before the queried time.This left-to-right restriction ensures the model is generative.Source: §3 · p. 3
- Training maximizes the autoregressive point-process log-likelihood, increasing observed-event log intensities while penalizing intensity assigned to events that did not occur.The integral term is approximated with Monte Carlo methods during training.Source: §3 · pp. 3–4
- Computing intensity is costly because each possible event at each queried time receives a deep embedding, and prediction procedures evaluate many possible events.Appendix A describes a speedup that shares embeddings among similar events, though selective attention can require different embeddings.Source: §3 · p. 4
4 MULTI-HEAD SELECTIVE ATTENTION
Selective attention constrains which past event types can influence a possible future event. Domain experts supply these influence rules, and each rule receives its own attention parameters.
4 MULTI-HEAD SELECTIVE ATTENTION
- For Frank’s emails, the example excludes Eve’s earlier emails to Adam, while Eve’s emails may depend on both event types.The corresponding allowed edges are ffff <- ffff, eeee <- eeee, and eeee <- ffff.Source: §4 · p. 4
- A domain expert specifies which event types can influence which others; the model does not learn this sparsity pattern from sequence data.The specified edges form an influence, or Granger-causality, matrix and may reduce the training data needed.Source: §4 · p. 4
- Each influence rule gets a separate attention head with its own parameters, queries, keys, and values, so it can retrieve distinct information from the past.For example, Eve can attend separately to her own emails and Frank’s emails when deciding whether to email Adam.Source: §4 · p. 4
- A rule’s history contains only past events it makes visible, and its contribution is small or absent when no such events are present or relevant.The rule-specific attention weights and value projections determine how visible events contribute.Source: §4 · p. 4
5 ATTENTIVE NEURAL DATALOG THROUGH TIME
Attentive NDTT replaces recurrent fact embeddings with Transformer-style attention while retaining a symbolic database whose rules update facts over time. It embeds facts according to their provenance and trains possible-event embeddings to predict event likelihood.
5 ATTENTIVE NEURAL DATALOG THROUGH TIME
- A Datalog through time program updates a database as events occur, and the resulting facts determine which event types can happen next.Rules can add or remove facts, while deductive rules derive facts from those currently in the database.Source: §5 · p. 5
- An NDTT program augments a DTT program with dimensionality declarations; rules that add facts also compute or update their vector embeddings.The database’s embedding dimension can change as facts enter or leave, while the program’s fixed rules determine its parameter count.Source: §5 · p. 5
- Each fact embedding represents when and how the fact was established, using learned parameters associated with the rules that deduced or added it.This symbolic database handles keeping many boolean facts current, including when relevant events may be rare.Source: §5 · pp. 5–6
- Attentive NDTT replaces LSTM-style recurrence with Transformer-style attention to compute embeddings for facts in a neural Datalog through time program.The method maps an NDTT program to a neural architecture while preserving its symbolic rules.Source: §5 · p. 6
- For add rules, attention uses the entire triggering condition, not just the event that triggered the addition, as the attended item.Its attention key and value are defined for that combined condition.Source: §5 · p. 6
- For deduction rules, the embedding sums pooled representations of matching fact tuples; learned softmax pooling returns zero when there are no matches.This deduction step follows the prior NDTT formulation rather than using attention.Source: §5 · p. 6
- In the forum example, rules let each employee’s messages affect the shared forum with O(K) rules, rather than O(K^2) direct employee-to-employee rules.A variable-based extension can govern all K employees with O(1) rules.Source: §5 · p. 7
- Facts change discretely, but their embeddings evolve continuously, and embeddings of possible events are trained to predict how likely those events are.NDTT rules allow events to influence embeddings of later facts, while deduction rules allow contemporaneous facts to influence one another.Source: §5 · p. 7
6 RELATED WORK
Prior multivariate point-process work spans several application areas, and many neural models use recurrent architectures. Earlier attention-based point-process models use a different approach to computing event intensities than the one described here.
6 RELATED WORK
- Multivariate point processes have been applied to document streams, Granger-causality learning, network analysis, recommendation, and social-network analysis.The cited work includes He et al. (2015), Du et al. (2015a), Xu et al. (2016), Zhang et al. (2020b; 2021), Choi et al. (2015), Etesami et al. (2016), Du et al. (2015b), Guo et al. (2015), and Lukasik et al. (2016).Source: §6 · p. 7
- Many neural point-process models use LSTMs and have outperformed classical models such as the Hawkes process on many real-world datasets.Examples include Du et al. (2016), Mei & Eisner (2017), Xiao et al. (2017a;b), and Mei et al. (2020a).Source: §6 · p. 7
- Earlier attention-based point-process models derive intensity from the latest actual event’s embedding, whereas this paper derives it from an embedding of the possible event at time t.Zuo et al. (2020) and Zhang et al. (2020a) introduced Transformer Hawkes and self-attentive Hawkes processes; Zhu et al. (2021) independently proposed the possible-event approach but did not evaluate it experimentally.Source: §6 · p. 7
7 EXPERIMENTS
The experiments compare attention-based event models with recurrent and Transformer baselines on synthetic and real-world data. A-NHP is competitive across the benchmark tasks, while A-NDTT achieves comparable accuracy to NDTT on RoboCup and shows improvements over NHP on IPTV.
7 EXPERIMENTS
- The evaluation measures held-out log-likelihood and next-event time and type prediction on synthetic and real-world datasets.The real-world A-NHP comparison covers MIMIC-II clinical visits and StackOverflow user awards.Source: §7 · p. 7, §7.1 · p. 9
7.1 COMPARISON OF DIFFERENT TRANSFORMER ARCHITECTURES
- A-NHP computes each event intensity by attending over all previous events with attention weights independent of event type.It uses a single coarse event type to speed intensity computation.Source: §7.1 · p. 8
- In synthetic pilot tests, NHP, SAHP, and A-NHP perform similarly across four datasets and generally outperform THP, especially on event timing.A-NHP remained satisfactory even when its model was misspecified.Source: §7.1 · p. 8
- On MIMIC-II, A-NHP is a co-winner on every task; on StackOverflow, it wins two of three tasks and ties NHP on the third.A-NHP also took a fraction of NHP’s training time when sequences were reasonably long.Source: Figure 2 · p. 8, §7.1 · p. 9
7.2 A-NDTT VS. NDTT
- On RoboCup, rule-based NDTT and A-NDTT substantially improve log-likelihood and event-type error over unstructured models, with no significant difference between NDTT and A-NDTT.The comparison uses the same NDTT program and rules, replacing its LSTM architecture with a continuous-time Transformer in A-NDTT.Source: §7.2 · p. 9, Figure 3 · p. 9
- On IPTV, rules and attention each significantly improve NHP log-likelihood, while rules reduce event-type error less under A-NDTT than under NDTT.Figure 3 reports evaluation on RoboCup and IPTV using the Figure 2 evaluation methods; training used log-likelihood.Source: Figure 3 · p. 9, §7.2 · p. 10
8 CONCLUSION
The paper generalizes Transformers to discrete events in continuous time and integrates the architecture with NDTT. The resulting models are competitive, and the attention-based NDTT is simpler and shallower.
8 CONCLUSION
- The architecture builds embeddings of actual and possible events at any time from event representations and context, and can predict intensity while supporting parallel log-likelihood computation.On multiple real-world datasets it outperforms other Transformer-based models, and on multiple synthetic datasets it beats or ties them.Source: §8 · p. 10
- A-NDTT integrates attention with NDTT, which derives neural models from logic programs, and achieves competitive performance with a simpler, shallower architecture.The paper makes its code and datasets available.Source: §8 · p. 10
Appendices
The appendices explain architectural choices for attention, time embeddings, and computation, then extend an NDTT example with variables to represent larger systems compactly.
A DISCUSSION OF ARCHITECTURAL DETAILS
- The Transformer equations omit LayerNorm and feed-forward steps after preliminary experiments found no benefit, though the code retains them as options.The attention denominator includes an extra 1 so embeddings remain close to their previous-layer values when history is weakly relevant.Source: A DISCUSSION OF ARCHITECTURAL DETAILS · p. 14
- Sinusoidal time embeddings use data-derived wavelength bounds to accommodate different dataset time scales, with the shortest training-event gap setting m and the observation horizon bounding M.The method concatenates time embeddings at every layer for direct access to time information.Source: A DISCUSSION OF ARCHITECTURAL DETAILS · p. 14
- Computing a full embedding for every possible event is expensive; grouping events with the same relevant history allows their coarse event embedding to be shared for intensity computation.When an event occurs, its fine-grained embeddings still must be computed; without selective attention, one coarse type can reduce runtime by a factor of |E|.Source: A DISCUSSION OF ARCHITECTURAL DETAILS · pp. 14–15
- Attention-head outputs are summed because rules can have different numbers of heads, avoiding variable-dimensional concatenation projections while retaining expressive power.The appendix also notes the extra denominator term acts like attention to a fixed zero-valued dummy object.Source: A DISCUSSION OF ARCHITECTURAL DETAILS · p. 15
B NDTT EXAMPLE WITH VARIABLES
- Capitalized variables let NDTT rules generalize the two-user, one-forum example to many employees and team forums using a constant number of rules and parameters.Rules can also encode membership so messages affect a forum and its members’ subsequent embeddings.Source: B NDTT EXAMPLE WITH VARIABLES · pp. 15–16, Figure 4 · p. 16
- For a rule with variables, attention ranges over its ground instantiations whose facts were true and whose triggering event occurred at the relevant time.Rule-specific parameters are shared across instantiations unless the program specifies otherwise.Source: B NDTT EXAMPLE WITH VARIABLES · p. 16
C PARAMETER DIMENSIONALITY SPECIFICATION FOR A-NDTT
The appendices specify how A-NDTT dimensions follow fact types and rules, and describe likelihood estimation and its Monte Carlo approximation.
C PARAMETER DIMENSIONALITY SPECIFICATION FOR A-NDTT
- A fact’s type is determined by its functor, and the NDTT program declares a shared embedding dimensionality for facts of each type.These choices determine deduction-rule parameter dimensions; add-rule value vectors match the head-fact embedding dimension, while key and query dimensions may be overridden.Source: C PARAMETER DIMENSIONALITY… · p. 17
D LIKELIHOOD COMPUTATION DETAILS
- The log-likelihood rewards observed-event intensities and penalizes total intensity over times when no event occurred.Its integral term is approximated by Monte Carlo sampling of times; training uses as many samples as observed events, while tuning and final reporting use ten times as many.Source: D LIKELIHOOD COMPUTATION DETAILS · p. 17
- Summing intensities over all event types at each sampled time can be expensive; A-NDTT uses the downsampling method from Mei et al. (2020a) because this computation is difficult to parallelize.Source: D LIKELIHOOD COMPUTATION DETAILS · p. 17
E HOW TO PREDICT EVENTS
The section explains how the models sample future events and how the paper evaluates next-event time and type predictions. It also points to dataset statistics without reporting their values here.
E HOW TO PREDICT EVENTS
- Exact sampling uses thinning, enabled by a bounded event intensity whose upper bound can be computed with interval arithmetic.The sinusoidal time embedding keeps its input in the compact set [−1, 1]^d, which supports the boundedness argument.Source: E HOW TO PREDICT EVENTS · p. 17
- Evaluation predicts each held-out event’s time from preceding events and its type from the true time plus preceding events.Time is scored by RMSE from average L2 loss; type is scored by error rate from average 0-1 loss.Source: E HOW TO PREDICT EVENTS · p. 18
- Minimum Bayes risk selects the time and type predictions with the lowest expected loss.The time estimate is the mean under the event-time density; the type is the most probable type at that predicted time.Source: E HOW TO PREDICT EVENTS · p. 18
F.1 DATASET DETAILS
- Table 1 is identified as a summary of statistics for each dataset, but this passage gives no values.Source: §F.1 · p. 18, Table 1 · p. 18
F.1.1 PILOT EXPERIMENTS ON SIMULATED DATA
The pilot compares four point-process models across four synthetic datasets, finding close performance for NHP, SAHP, and A-NHP. The appendix also identifies the data sources and implementation bases for the experiments.
F.1.1 PILOT EXPERIMENTS ON SIMULATED DATA
- The pilot draws data from randomly initialized models and fits all four models to each of four synthetic datasets.Each model uses 800 training, 100 validation, and 100 test sequences; sequence length is sampled from Uniform(49, 99).Source: §F.1.1 · p. 18
- NHP, SAHP, and A-NHP perform very similarly across four synthetic datasets, generally outperforming THP, especially on timing prediction.THP may fit timing poorly when misspecified because its intensity family imposes particular temporal-shape constraints; Figure 5 compares held-out log-likelihood across the distribution families.Source: §F.1.1 · p. 18, Figure 5 · p. 19
F.1.2 OTHER DATA DETAILS
- MIMIC-II and StackOverflow use versions processed by Du et al. (2016), while RoboCup uses the version processed by Chen & Mooney (2008).Source: §F.1.2 · p. 19
F.2 IMPLEMENTATION DETAILS
- The NHP and NDTT implementations build on their prior public code, with A-NHP and A-NDTT built on those respective implementations; THP and SAHP also use public code.The NHP code was modified for PyTorch 1.7, and A-NHP has a standalone GPU implementation.Source: §F.2 · p. 19
F.3 TRAINING DETAILS
Action-wise results show that A-NDTT and NDTT differ on particular RoboCup actions, while reruns preserve Figure 6’s qualitative pattern. The appendix also details training configurations and a possible route to GPU parallelism.
F.3 TRAINING DETAILS
- Model dimensionality D and attention depth L were tuned on the development set for each model, dataset, and training size.The search used D ∈ {4, 8, 16, 32, 64, 128} and L ∈ {1, 2, 3, 4, 5, 6}; training used Adam and early stopping on development log-likelihood.Source: §F.3 · pp. 19–20
- On RoboCup, NDTT and A-NDTT took 62 and 149 seconds per sequence, respectively; NHP and A-NHP took 66 and 95 seconds.NDTT/A-NDTT ran on CPUs; NHP/A-NHP also ran on CPUs because the sequences were too long for the K80 GPU memory.Source: §F.3 · p. 20
F.4 TRAINING PARALLELISM
- A proposed GPU-training procedure first computes database facts on CPU, then builds an O(|R|·I) graph to compute fact embeddings across times.Embeddings can be parallelized across sequences, times, and facts sharing a rule, but masking leaves GPU work idle; the approach is most promising when much computation remains unmasked.Source: §F.4 · pp. 20–21
F.5 MORE RESULTS
- A-NDTT improves data fitting for kickoff events, while NDTT is better for other actions; A-NDTT improves time prediction for goals, kickoffs, and passes.For participant prediction, A-NDTT is significantly better on kickoffs, with no difference on other actions; both perform perfectly on goals and kicks.Source: Figure 6 · p. 21, §F.5 · p. 21
- Figure 6’s action-wise results do not change qualitatively when rerun with different random seeds.Figure 7 presents one replication per row.Source: §F.5 · p. 21, Figure 7 · p. 22