Source-linked AI summary
TimeSHAP: Explaining Recurrent Models through Sequence Perturbations
João Bento, Pedro Saleiro, André F. Cruz, Mário A. T. Figueiredo, Pedro Bizarro
TL;DR
RNN predictions depend on sequence history, but existing explainers provide limited recurrent attribution. TimeSHAP extends KernelSHAP with model-agnostic sequence perturbations and feature-, event-, and cell-level explanations, using pruning to reduce cost and variance. In a real-world account-takeover model, its explanations aligned with analyst-identified fraud cues and exposed potentially discriminatory reliance on client age.
Problem
Existing explainers do not adequately attribute RNN predictions to prior events and sequence-wide features, despite RNNs being used in sequential decision-making and high-stakes fraud detection.
Method
TimeSHAP is a model-agnostic, post-hoc recurrent explainer that extends KernelSHAP with sequence perturbations, feature-, event-, and cell-level attributions, and temporal coalition pruning.
Results
41% of the sequence’s score was attributed on average to the event being explained, while explanations identified analyst-confirmed account-takeover cues and age-related reasoning later confirmed in a bias audit.
Takeaways & Limitations
TimeSHAP supports model validation and human understanding of recurrent predictions while revealing features and events relevant to fraud decisions.
Takeaways & Limitations
Cell-level explanations can become costly because the number of cells equals the product of events and features.
Abstract
from arXiv · showhide
Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agnostic recurrent explainer that builds upon KernelSHAP and extends it to the sequential domain. TimeSHAP computes feature-, timestep-, and cell-level attributions. As sequences may be arbitrarily long, we further propose a pruning method that is shown to dramatically decrease both its computational cost and the variance of its attributions. We use TimeSHAP to explain the predictions of a real-world bank account takeover fraud detection RNN model, and draw key insights from its explanations: i) the model identifies important features and events aligned with what fraud analysts consider cues for account takeover; ii) positive predicted sequences can be pruned to only 10% of the original length, as older events have residual attribution values; iii) the most recent input event of positive predictions only contributes on average to 41% of the model's score; iv) notably high attribution to client's age, suggesting a potential discriminatory reasoning, later confirmed as higher false positive rates for older clients.
1 INTRODUCTION
RNN explanations must account for both current inputs and sequence history, yet existing SHAP methods focus on a single input vector. TimeSHAP adapts KernelSHAP for recurrent sequences and validates its explanations on account-takeover fraud detection.
- RNN predictions depend on immediate inputs, prior events, and hidden-state context, which single-vector explainers fail to attribute.
- High-stakes fraud detection requires explanations that are model-agnostic, post-hoc, and compatible with inference-API access.
- TimeSHAP extends KernelSHAP to recurrent sequences, attributing importance to features and previous events.
- Temporal coalition pruning aggregates older events, reducing computational cost and improving attribution stability.
- In a real-world account-takeover model, explanations matched analysts’ enrollment-login-transaction cues, while client age showed potentially discriminatory reasoning confirmed by a bias audit.
2 RELATED WORK
Related work includes model-specific and model-agnostic explainers, but KernelSHAP does not directly capture recurrent sequence history. TimeSHAP addresses this through sequence-wide perturbations rather than current-instance-only or hidden-state-only attribution.
- Attention and gradient-based methods explain RNNs using architecture or derivatives, but attention’s explanatory status remains controversial.
- Model-agnostic explainers use post-hoc predictions under input perturbations and can support varied model architectures.
- KernelSHAP satisfies local accuracy, missingness, and consistency, but its perturbations ignore recurrent hidden state and sequence history.
- KernelSHAP approximates a complex model locally with a linear feature-importance model and distributes the original score using Shapley values.
- Alternative recurrent adaptations either explain only current features or assign the entire previous sequence one hidden-state attribution, whereas TimeSHAP uses sequence-wide perturbations.
3 TIMESHAP
TimeSHAP extends KernelSHAP to recurrent sequences, producing feature-, event-, and cell-level attributions while using pruning and grouping to control the exponential cost of explaining long inputs.
- Sequence perturbations: TimeSHAP provides feature and event attributions throughout a sequence while preserving local accuracy, missingness, and consistency.Its linear explainer approximates the recurrent model’s local behavior, with the attribution dimension set to features or events.
- Sequence perturbations: Explanations perturb either feature rows or event columns by replacing toggled-off values with an uninformative background matrix.Feature perturbations toggle a feature across all events, whereas event perturbations toggle all features of one event.
- Temporal coalition pruning: The number of temporal coalitions grows exponentially with sequence length, motivating temporal coalition pruning for arbitrarily long recurrent inputs.The pruning algorithm scans sequence splits and identifies an older prefix whose aggregate importance falls below threshold η.
- Temporal coalition pruning: Pruning reduces TimeSHAP’s runtime from O(2^l) to O(2^(l−i)), while sacrificing granularity for grouped older events and retaining individual treatment of newer events.The pruning algorithm itself scales linearly with the number of events, O(l).
- Cell-level explanations: Cell-level explanations address O(2^(ld)) complexity by grouping cells and isolating cells at intersections of relevant features and events.Feature or event relevance is determined by an absolute attribution exceeding the user-defined threshold θ.
4 CASE STUDY
The case study applies TimeSHAP to a real-world GRU-based account-takeover detector, showing that pruning makes explanations more tractable while event, feature, and cell attributions reveal concrete fraud cues and a possible age-related bias.
- The evaluated GRU-based model achieved 84.3% recall at 1% false positive rate on validation and 79.9% recall at 0.89% false positive rate on test data.
- Pruning Results: 58.3% of input samples permitted exact Shapley values at pruning tolerance η=0.025, which balanced consistency, run-time, and granularity with RSD 0.98.Without pruning, the relative standard deviation was 1.71; η=0.05 reduced it to 0.68 but produced coarser explanations.
- Global Explanations: The latest event contributed 41% of the sequence score on average, while preceding events contributed 59%, including distant events that retained meaningful attribution.Globally, the latest transaction had average contribution 0.28, while events before index −5 averaged around zero but sometimes retained high contributions through indices −5 to −20.
- Global Explanations: Feature explanations emphasized transaction type, event type, client age, IP, and location, while age attribution was followed by a disparity in false positive rates for older clients.For sequence A, transaction amount, IP feature D, and client age were most relevant; sequence B also showed location-related discrepancies.
- Pruning Results: Older events had sharply declining aggregate importance, allowing sequence A to be pruned to 11 events and sequence B to 9 events at η=0.025.The pruned representations grouped 36 older events for sequence A and 277 older events for sequence B.
- Local explanations: Local explanations identified transaction events at t=0 and t=−4 as crucial in sequence A, with Shapley values 0.36 and 0.17, respectively.The high transaction amount at t=0 accounted for 0.21 of the score, while non-relevant events collectively accounted for 0.005.
- Local explanations: Sequence B’s enrollment-login-transaction pattern was sufficient for correct identification before the current transaction, whose contribution was near null at 0.001.The most relevant events were t=−4 and t=−1, with Shapley values 0.48 and 0.24.
- Local explanations: Cell-level explanations localized importance to the intersection of the most relevant event and feature, whose importance was 0.274.The next reported cell groups had relevance 0.153 for event t=−4 and an incomplete reported value for event t=−1.
5 CONCLUSION
TimeSHAP is a model-agnostic, post-hoc explainer for recurrent models that provides event-, feature-, and cell-level attributions, with pruning to reduce execution time and explanation variance. In a real-world account-takeover model, its explanations aligned with expert-identified fraud patterns while revealing uneven temporal contributions and a potential age-related bias.
- TimeSHAP explains recurrent-model predictions through event-, feature-, and cell-level attributions using sequence-tailored perturbations.It is designed for recurrent models of any architecture when features and an inference API are available.
- The pruning algorithm decreases TimeSHAP’s execution time and attribution variance, supporting more practical explanation workflows.The authors connect these explanations to model validation, debugging, development iteration, and human review.
- In a real-world account-takeover model, domain experts corroborated the important features and events as the typical enrollment-login-transaction pattern.This pattern was used to assess the reliability of the model’s explanations.
- 41% of the sequence’s score was attributed on average to the event being explained, while preceding events represented 59%.The result indicates that prior sequence context contributed substantially alongside the current event.
- The client’s age was an important feature, suggesting potentially discriminatory reasoning that a later bias audit confirmed.The conclusion identifies age-related attribution as a concern in the examined fraud-detection model.