PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Deep Learning Interpretability in Time Series Predictions

Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, Soheil Feizi

arXiv:2010.13924v1cs.LGstat.ML

TL;DR

Saliency methods are widely used for explanations, but their reliability for multivariate time series is comparatively underexplored. This paper benchmarks them across architectures and synthetic datasets, finding broad failures caused by conflated time and feature domains, while TSR substantially improves saliency maps.

  • Problem

    Reliable saliency explanations for multivariate time series remain comparatively underexplored despite the importance of interpreting models used with time-series data.

  • Method

    The paper systematically evaluates gradient- and perturbation-based saliency methods across recurrent, convolutional, and Transformer architectures using synthetic time-series datasets with known informative features.

  • Results

    Saliency methods and architectures generally fail to reliably identify feature importance over time in multivariate time series, with performance often similar to random saliency.

  • Takeaways & Limitations

    Time and feature domains should be decoupled when interpreting multivariate time series, and the proposed two-step TSR approach substantially improves saliency-map quality.

  • Takeaways & Limitations

    Temporal Saliency Rescaling increases computational complexity because it performs multiple gradient calculations.

Abstract

from arXiv · show

Saliency methods are used extensively to highlight the importance of input features in model predictions. These methods are mostly used in vision and language tasks, and their applications to time series data is relatively unexplored. In this paper, we set out to extensively compare the performance of various saliency-based interpretability methods across diverse neural architectures, including Recurrent Neural Network, Temporal Convolutional Networks, and Transformers in a new benchmark of synthetic time series data. We propose and report multiple metrics to empirically evaluate the performance of saliency methods for detecting feature importance over time using both precision (i.e., whether identified features contain meaningful signals) and recall (i.e., the number of features with signal identified as important). Through several experiments, we show that (i) in general, network architectures and saliency methods fail to reliably and accurately identify feature importance over time in time series data, (ii) this failure is mainly due to the conflation of time and feature domains, and (iii) the quality of saliency maps can be improved substantially by using our proposed two-step temporal saliency rescaling (TSR) approach that first calculates the importance of each time step before calculating the importance of each feature at a time step.

1 Introduction

The paper benchmarks saliency methods for multivariate time series using synthetic datasets with known informative features, finding failures in feature attribution and proposing TSR to improve maps.

  • Benchmark and scope: The study compares perturbation- and gradient-based interpretability methods across recurrent, convolutional, and Transformer architectures for multivariate time-series classification.Synthetic datasets capture temporal and feature-level signal variation, with informative feature positions known a priori.
  • Evaluation: Evaluation progressively masks features ranked as important and measures model-accuracy degradation, then computes precision and recall against known informative features.This distinguishes whether selected features contain meaningful signals from whether all signal-bearing features are identified.
  • Findings: Saliency methods that work well for images often fail to provide similarly reliable interpretations for multivariate time series.The reported observations also indicate that architectures significantly affect saliency-map quality.
  • Proposed approach: The paper proposes Temporal Saliency Rescaling, which first estimates time-step relevance and then feature relevance within sufficiently relevant time steps.The final importance score is the product of the associated time and feature relevance scores, and the approach can be applied on top of existing methods.

2 Background and Related Work

Prior work develops diverse feature-attribution methods and modification-based evaluations, while this paper provides a systematic multivariate time-series benchmark across architectures and proposes an adaptation method.

  • Interpretability research: Interpretability research includes model-design approaches, input-feature importance estimators, and human-concept explanations.The common goal is understanding how a network makes predictions, while reliability remains a central question.
  • Evaluation methods: Modification-based evaluations rank features by saliency, eliminate higher-ranked features recursively, and measure degradation in trained-model accuracy.Some approaches retrain the model after feature elimination.
  • Contribution: The paper presents the first systematic benchmark of saliency methods across multiple neural architectures in a multivariate time-series setting.It also identifies common limitations and proposes a solution for adapting existing methods to time series.
  • Saliency methods: The benchmark compares backpropagation- and perturbation-based saliency methods, using random assignment as a baseline control.Included methods span gradient, integrated-gradient, smoothing, reference-based, occlusion, ablation, permutation, and Shapley-value approaches.
  • Architectures: Three architecture groups are investigated: recurrent networks, CNNs such as TCNs, and Transformers.The recurrent group includes LSTM variants, while TCNs are used for long-sequence time series.

3 Problem Definition

The problem is multivariate time-series classification in which every time step contributes to the final output, and saliency assigns relevance to each feature at each time.

  • Input and output: The network receives a multivariate input X with N features across T time steps and produces class outputs after the sequence.Labels are available after the last time step, while all time steps contribute to the classification.
  • Saliency representation: For a target class c, a saliency method returns an N×T relevance matrix assigning score R_i,t(X) to feature i at time t.The formulation explicitly represents importance across both feature and temporal dimensions.

4 Benchmark Design and Evaluation Metrics

The benchmark uses controlled synthetic datasets and masking-based evaluation to assess whether saliency methods identify informative features across time and feature dimensions.

  • Dataset design: Synthetic datasets vary how discriminating signals are distributed across time and features, including differences in value, position, and shape.The datasets also cover multiple time-series generation processes and yield 70 datasets in total.
  • Metric design: The evaluation addresses the assumption that saliency rankings faithfully represent importance and the distribution shift caused by feature elimination.Percentage-based removal addresses ranking differences, while synthetic-distribution masking preserves the i.i.d. setting.
  • Masking evaluation: Modification-based metrics are evaluated by ranking relevance, masking selected features with values from the original distribution, and measuring accuracy degradation.The masking levels range from d = 0 to 100 in increments of 10.
  • Interpretation: Masking can sharply reduce accuracy when a removed feature is necessary, but unchanged accuracy may reflect either incorrect attribution or insufficient removal.Some architectures may use more feature information when making predictions.
  • Metrics: The benchmark reports weighted precision and recall, including area under precision, recall, and precision-recall curves across degradation levels.It also evaluates feature- and time-level precision and recall and compares against a random baseline.

5 Saliency Methods Fail in Time Series Data

Across synthetic time-series experiments, saliency methods often identify informative times without reliably isolating informative features, with performance strongly affected by architecture and inconsistent across methods.

  • Most architecture–method pairs fail to identify feature importance over time, including on synthetic datasets and time-series MNIST despite accurate classification.
  • Across architectures, no consistent saliency-method trend appears; TCN Grad and SmoothGrad drop accuracy steeply, whereas LSTM shows little distinction from random assignment.
  • Architecture has the largest effect on precision and recall, while results show no clear distinctions between saliency methods.
  • Methods can identify informative time steps while failing to identify informative features within those steps.
  • A steep accuracy drop does not establish correct feature identification because leftmost masking curves often have the lowest precision and recall.
  • The masking evaluation compares gradient-based and non-gradient methods against random rankings, using accuracy degradation to assess salient-feature selections.

6 Saliency Maps for Images versus Multivariate Time Series

Comparisons between image-style and multivariate time-series representations show that saliency maps are more interpretable when time and feature domains are not conflated.

  • CNN maps distinguish informative pixels, whereas TCN maps struggle to distinguish important features within an informative time step.
  • Although examined methods generally work for images, they generally fail for multivariate time series; treating time series as images is not always feasible because performance may decrease and dimensionality may increase.
  • CNN and univariate TCN produce interpretable maps, while bivariate and multivariate TCN maps are harder to interpret.
  • The reported failure occurs when saliency maps conflate time and feature domains; maps are relatively accurate when inputs use only one domain.
  • Figure 8 compares CNN with uni-, bi-, and multivariate TCN treatments using saliency maps for Middle Box and shared samples.

7 Temporal Saliency Rescaling

Temporal Saliency Rescaling (TSR) separates time relevance from feature relevance by masking time steps first, then features within selected steps. Experiments report improved saliency quality, with TSR + Gradient outperforming other methods on all benchmark metrics.

  • Temporal Saliency Rescaling: TSR first computes each time step’s relevance from the total saliency change caused by masking that step.The first stage identifies time steps whose relevance exceeds threshold α.
  • Temporal Saliency Rescaling: TSR then computes feature relevance within time steps whose time relevance exceeds threshold α.Restricting feature-level analysis to selected time steps reduces overall computational complexity.
  • Motivation: Most saliency methods identify informative time steps but fail to distinguish important from non-important features within those steps.TSR is designed to decouple time and feature importance scores.
  • Results: TSR improves saliency quality across architectures and interpretability methods except SmoothGrad.The paper attributes SmoothGrad’s exception probably to noise in its gradients making a noisy baseline inappropriate.
  • Results: TSR + Grad outperforms other methods on all reported benchmark metrics for the TCN experiments.Table 1 evaluates AUPR, AUP, AUR, and AUC on the Middle Box and Moving Box synthetic datasets; higher AUPR, AUP, and AUR are better, while lower AUC is better.

8 Summary and Conclusion

The paper benchmarks interpretability methods for multivariate time series using known informative features and accuracy degradation after masking salient features. It finds that common methods generally fail in this setting, while TSR substantially improves saliency-map quality.

  • Benchmark: The authors create a synthetic benchmark with known positions of informative features to quantify each interpretation-method and architecture pair.They evaluate interpretation quality by masking inferred salient features and measuring degradation in trained-model accuracy.
  • Evaluation: Precision and recall are calculated for each interpretation-method and architecture pair from the identified feature sets.These metrics assess whether selected features are informative and how many informative features are recovered.
  • Findings: Common gradient-based and perturbation-based saliency methods fail to produce high-quality interpretations for multivariate time series.Their performance is often similar to random saliency, with no clear distinction across methods on multiple metrics.
  • Findings: Saliency methods generally identify salient time steps but cannot distinguish important from non-important features within a time step.The paper identifies conflation of temporal and feature domains as the central breakdown, while noting that the exact mathematical mechanism remains open.
  • Conclusion: The proposed two-step temporal saliency rescaling approach substantially improves saliency-map quality in time series applications.It adapts existing saliency methods by separating temporal and feature relevance.

9 Broader Impact

The paper frames reliable interpretation as necessary for serious DNN adoption in fields such as Neuroscience, Medicine, and Finance. It emphasizes that incorrect explanations can be worse than providing no interpretation.

  • Motivation: Meaningful interpretation of DNNs is presented as a technical barrier to serious adoption in Neuroscience, Medicine, and Finance.Accurate DNN predictions alone are not sufficient for routine use in high-stakes applications such as healthcare.
  • Potential benefits: Interpretability can increase model transparency and ease application across research areas.Understanding model decisions can help guide modifications intended to produce better and fairer results.
  • Risks: Failure to provide faithful interpretations is described as a severe negative outcome.The paper states that, in many situations, having no interpretation is better than trusting an incorrect one.

Supplementary Material

The supplementary material details the saliency methods, synthetic-data design, realistic-data applications, and evaluation procedures used to benchmark interpretability across time-series settings.

  • Saliency methods: The benchmark compares backpropagation-based and perturbation-based methods that assign feature relevance at each time step.The listed methods include Gradient, Integrated Gradients, SmoothGrad, DeepLIFT, Gradient SHAP, Deep SHAP, Feature Occlusion, Feature Ablation, Feature Permutation, and Shapley Value Sampling, with random as a control.
  • Saliency methods: Perturbation methods estimate attribution by masking, replacing, or permuting input features and measuring the resulting output change.Feature Occlusion uses contiguous regions, while Feature Ablation and Feature Permutation can operate on individual or grouped features.
  • Synthetic datasets: The synthetic datasets vary informative features and informative time intervals to capture different temporal and feature configurations.Design levels include shape, signal movement, and positional temporal-versus-feature conditions.
  • Synthetic datasets: Synthetic data are generated from multiple processes, including Gaussian noise, harmonic, pseudo-periodic, autoregressive, continuous autoregressive, and NARMA sequences.The generation uses non-uniform time sampling and the TimeSynth package.
  • Additional datasets: The study also examines HCP fMRI task classification and multivariate Time Series MNIST as realistic or general test cases.For MNIST, each sample has 28 time steps and an embedding size of 28; HCP fMRI data are used to classify the task performed during scanning.
  • Evaluation: For synthetic data, informative features support precision and recall calculation, while saliency distributions reveal architecture- and method-specific patterns.Aside from Feature Ablation, saliency decays exponentially with feature ranking and distributions appear similar across methods for a given architecture.

Precision

The benchmark evaluates saliency methods using precision, recall, and accuracy degradation across synthetic time-series settings. Results indicate that saliency quality varies by architecture and method, while TSR generally improves maps except for Gradient SHAP and SmoothGrad.

  • Metrics: Weighted precision measures the saliency-weighted fraction of high-saliency features that are informative.The definition counts informative high-saliency features relative to all high-saliency features using saliency values rather than feature counts.
  • Metrics: Weighted recall measures the saliency-weighted fraction of informative features that receive high saliency.Precision and recall are evaluated across different levels of model accuracy degradation.
  • Metrics: The experiments report area under precision, recall, and precision–recall curves, alongside feature- and time-level precision and recall.Curves are computed from precision and recall values at different degradation levels, with a permuted-saliency random baseline.
  • Temporal Saliency Rescaling: TSR first scores time-step relevance, then scores feature relevance within time steps whose relevance exceeds a threshold.Feature grouping provides a TSR variation, while the approach increases complexity because it performs multiple gradient calculations.
  • Results: Applying TSR produces a definite saliency-quality improvement across architectures and methods, except for Gradient SHAP and SmoothGrad.The improvement is reported for time-series MNIST and synthetic datasets, while the evaluation figures cover accuracy drop, precision, and recall across dataset variants.
  • Results: For bivariate and multivariate time series, Grad maps are harder to interpret, whereas TSR significantly improves them and can surpass image or univariate representations.The comparison spans CNN image treatment and univariate, bivariate, and multivariate TCN treatments.
Loading 2010.13924v1…