Source-linked AI summary
Interpretable and Steerable Sequence Learning via Prototypes
Yao Ming, Panpan Xu, Huamin Qu, Liu Ren
TL;DR
Deep sequence models can be accurate yet difficult to explain, limiting transparency in applications that require understandable decisions. ProSeNet addresses this by combining RNNs with sparse, human-refinable prototypes for case-based predictions. Across several real-world tasks, it achieves comparable state-of-the-art performance, while user studies find prototypes aligned with human knowledge and refinement improves interpretability without performance loss.
Problem
Deep sequence models often act as black boxes, making their rationales difficult to explain and limiting transparency for critical decision-making.
Method
ProSeNet combines RNNs with prototype learning, using a few exemplar sequences for prediction and constrained training to incorporate user-specified prototypes.
Results
Across four real-world sequence prediction or classification tasks, ProSeNet achieves performance comparable to state-of-the-art methods while providing interpretability.
Takeaways & Limitations
User studies show that selected prototypes align with human knowledge, and interactive refinement improves interpretability without loss of performance.
Takeaways & Limitations
Prototype optimization is exponentially costly in sequence length before approximation, and steering assumes user knowledge can be expressed as input-output patterns.
Abstract
from arXiv · showhide
One of the major challenges in machine learning nowadays is to provide predictions with not only high accuracy but also user-friendly explanations. Although in recent years we have witnessed increasingly popular use of deep neural networks for sequence modeling, it is still challenging to explain the rationales behind the model outputs, which is essential for building trust and supporting the domain experts to validate, critique and refine the model. We propose ProSeNet, an interpretable and steerable deep sequence model with natural explanations derived from case-based reasoning. The prediction is obtained by comparing the inputs to a few prototypes, which are exemplar cases in the problem domain. For better interpretability, we define several criteria for constructing the prototypes, including simplicity, diversity, and sparsity and propose the learning objective and the optimization procedure. ProSeNet also provides a user-friendly approach to model steering: domain experts without any knowledge on the underlying model or parameters can easily incorporate their intuition and experience by manually refining the prototypes. We conduct experiments on a wide range of real-world applications, including predictive diagnostics for automobiles, ECG, and protein sequence classification and sentiment analysis on texts. The result shows that ProSeNet can achieve accuracy on par with state-of-the-art deep learning models. We also evaluate the interpretability of the results with concrete case studies. Finally, through user study on Amazon Mechanical Turk (MTurk), we demonstrate that the model selects high-quality prototypes which align well with human knowledge and can be interactively refined for better interpretability without loss of performance.
1 INTRODUCTION
ProSeNet addresses the interpretability gap in deep sequence models by using prototypes for case-based reasoning and user steering. It targets broad sequence applications while maintaining performance comparable to state-of-the-art methods.
- Motivation: RNN-based sequence models perform well but are often black boxes, motivating models with inherent interpretability for critical decisions.Post-hoc explanations may incompletely or inaccurately capture the original model’s reasoning.
- Approach: ProSeNet combines an RNN backbone with prototype learning, making predictions from input similarity to prototypes in latent space.The RNN captures temporal structure while prototypes provide case-based explanations.
- Approach: Prototype construction favors simplicity, diversity, and sparsity so explanations contain concise, distinct, and few active exemplars.Prototypes may be short subsequences containing key events rather than complete noisy sequences.
- Steering: User-specified prototypes can be incorporated through constrained training, allowing domain experts to steer the model using their intuition and experience.The training procedure iteratively performs gradient descent and prototype projection.
- Evaluation: Experiments span automotive predictive maintenance, protein sequences, ECG signals, and customer-review sentiment analysis, demonstrating broad application coverage.The reported evaluation includes classification accuracy and interpretability studies.
- Evaluation: ProSeNet achieves comparable state-of-the-art performance while providing analogy-based interpretability, including after prototype refinement.The paper reports better interpretability without loss of performance when users refine prototypes.
2 RELATED WORK
Related work contrasts post-hoc explanations with inherently interpretable sequence models. ProSeNet extends sparse prototype-based reasoning from other settings to sequential prediction and evaluates interpretability through user studies.
- Post-hoc explanation: Post-hoc sequence explanations visualize hidden states, extract token importance, or distill models into simpler representations.These methods explain pre-existing black-box models rather than changing their internal design.
- Inherent interpretability: Post-hoc explanations may not precisely fit the original model, while traditional transparent models often struggle with complex temporal dependencies.This motivates sequence models designed with interpretability built in.
- Prototype reasoning: ProSeNet uses a few simplified prototypes instead of memorizing all nearest-neighbor cases, providing concise data summaries and sparse explanations.The approach also supports interactive prototype updates by human experts.
- Evaluation: The paper notes that no universal interpretability metric exists and evaluates whether prototypes explain predictions through user studies.Quantitative sparsity or complexity measures may not correspond directly to human interpretability.
3 METHODOLOGY
ProSeNet combines a recurrent sequence encoder, prototype-based similarity reasoning, and interpretable regularization to produce predictions explained by representative sequences. Training also projects prototypes onto observed subsequences and supports expert refinement for interpretability.
- 3.1 ProSeNet Architecture: Predictions are explained through the input’s similarity to representative prototype sequences in the learned latent space.Prototype-based comparisons provide classification references and analogical explanations for new sequences.
- 3.1 ProSeNet Architecture: The architecture encodes each input sequence into a fixed-length embedding, compares it with k prototype vectors, and maps similarities to output predictions.The encoder may use LSTM, Bi-LSTM, or GRU backbones; the prototype layer scores similarity in latent space.
- 3.2 Learning Objective: The objective combines cross-entropy with clustering, evidence, diversity, and sparsity-related regularization to balance accuracy and interpretability.Non-negative weights and an L1 penalty encourage more unitary and additive prototype semantics.
- 3.2 Learning Objective: Diversity regularization penalizes nearby prototypes, distributes them in latent space, and helps produce a sparser similarity vector.The experiments set the proximity threshold dmin to 1.0 or 2.0.
- 3.3 Optimizing the Objective: Prototype projection assigns each latent prototype to a closest training-sequence embedding, yielding meaningful observed prototype sequences without jointly training a decoder.Projection is performed every few epochs to reduce computational cost.
- 3.3 Optimizing the Objective: Brute-force prototype simplification has O(2^T N) complexity, so ProSeNet uses beam search with complexity O(w · T^2N) and w = 3 in the experiments.Beam search retains the w best candidates while iteratively removing events from candidate subsequences.
- 3.4 Refining ProSeNet with User Knowledge: Users can validate and update prototypes using domain knowledge to refine ProSeNet for better interpretability and performance.The interaction is intended to increase users’ understanding of the model and data, supporting trust.
4 EXPERIMENTAL EVALUATION
Experiments evaluate ProSeNet across vehicle diagnostics, sentiment analysis, protein classification, and ECG signals, combining accuracy comparisons with interpretability studies. Ablations examine prototype count, diversity regularization, and simplification, while human evaluation tests prototype quality and interactive steering.
- Experimental setup: ProSeNet is evaluated on four real-world sequence tasks using classification performance, qualitative interpretations, human studies, and ablations.The evaluation covers predictive maintenance, sentiment analysis, protein sequences, and ECG signals.
- Sentiment analysis: ProSeNet achieves similar, though slightly lower, accuracy than state-of-the-art bidirectional LSTMs on Yelp Reviews while learning interpretable representations.The comparison includes LSTM, Bi-LSTM, and ResNet baselines on validation and test sets.
- Protein classification: 97.0% average test accuracy on UniProtKB is slightly below Bi-LSTM at 97.4% and above ResNet at 96.7%.These results are averaged over 10 train-test splits, with standard deviations of 0.2%, 0.2%, and 0.3%, respectively.
- ECG classification: ECG prototypes capture clinically meaningful heartbeat patterns, including junctional escape beats and premature ventricular contractions, verified by two independent cardiologists.The model also has comparable performance to LSTM and slightly better accuracy than Residual CNN.
- Ablation studies: Accuracy increases rapidly with prototype count before flattening beyond 100 prototypes for UniProtKB and 40 for Yelp Reviews.The observed thresholds indicate diminishing performance gains as the prototype set grows.
- Ablation studies: Diversity regularization improves UniProtKB performance across prototype counts and makes Yelp prototype similarities sparser, more diagonal, and less redundant.Without regularization, heatmaps show near-duplicate prototypes; with λd = 0.01, prototypes are more diverse and evenly distributed.
- Ablation studies: Prototype simplification reduces average lengths without significantly changing accuracy: Yelp decreases from 20.1 to 15.1, and UniProtKB from 274.5 to 130.7.The length reductions apply to average prototype or subsequence lengths.
- Human evaluation: Human-guided refinement increases answer accuracy from 61.8% to 68.2%, while incremental training leaves test performance essentially unchanged, increasing it by 0.1%.The improved result is reported as 1.6% below human accuracy.
5 CONCLUSION AND FUTURE WORK
The paper presents ProSeNet as an interpretable and steerable deep sequence model combining prototype learning with RNNs. Across four real-world tasks, it matched state-of-the-art accuracy while providing interpretable prototypes that could be improved through human feedback.
- ProSeNet combines prototype learning with RNNs to achieve interpretability and high accuracy in deep sequence modeling.
- Experiments and case studies across four real-world sequence prediction and classification tasks found ProSeNet as accurate as other state-of-the-art machine learning techniques and more interpretable.
- A large-scale MTurk user study found that ProSeNet selected high-quality prototypes aligned with human knowledge for sentiment-analysis prediction and interpretation.
- Updating prototypes with user-study feedback improved interpretability without loss of performance, supporting a human-in-the-loop approach.
A PROTOTYPE SIMPLIFICATION VIA BEAM SEARCH
Prototype simplification uses beam search to retain candidate subsequences with the lowest Equation 3 scores. The procedure stops when further removal cannot improve the candidate set or no subsequences remain reducible.
- Beam search simplifies a prototype by evaluating candidate sequences with Equation 3 and retaining the w sequences with the lowest scores.
- The procedure returns the candidate set when fewer than w subsequences are available and terminates when subsequences are no longer reducible or no improvement exists.
- The algorithm takes an encoder, training data, a prototype, and beam width w as inputs, and outputs a projected prototype and subsequence.
- Candidate generation uses remove-one subsequences to search for simpler prototype representations.
B.1 Data Processing of Yelp Reviews
The Yelp Reviews preprocessing retained restaurant reviews, converted review text into word sequences, restricted human-evaluation reviews by length, and balanced classification classes by down-sampling.
- The dataset was restricted to reviews categorized as “Restaurant” according to the reviewed business’s category field.
- Review texts were tokenized into sequences of words using NLTK.
- For human evaluation, reviews containing more than 25 words were filtered out.
- Binary and fine-grained classification classes were balanced through down-sampling, with the largest class no more than twice the size of the smallest.
B.2 Post-processing of Human Evaluation Data
Human-evaluation questions were distributed across four questionnaires with quality checks. Responses failing quality checks or achieving below 50% accuracy were removed before computing human and model accuracy from majority votes.
- The 70 questions were divided evenly among four questionnaires containing 17 or 18 questions each to reduce participant burden.
- Each questionnaire included three quality-check questions, such as duplicates with reordered options or questions with obvious answers.
- Responses failing more than one quality-check question were filtered out.
- Responses with accuracy below 50% were further excluded before final accuracy calculations.
- Majority-voted options determined each question’s correct answer and supported the final human- and model-accuracy computations.
C.1 Sentiment Classification on Yelp Reviews
Figure 10 presents additional learned prototype sequences for Yelp Reviews, with simplified subsequences shown in bold uppercase text.
- The figure contains additional examples of learned prototype sequences from Yelp Reviews.
- Bold uppercase text identifies the simplified subsequences within the prototype sequences.
- The examples illustrate prototype representations for sentiment-review text data.
C.2 ECG Heartbeat Classification
Figure 11 lists the learned time-series prototypes for ECG heartbeats and overlays nearby test signals for comparison.
- The figure provides the complete list of time-series prototypes learned from ECG heartbeats.
- Bold lines represent the prototype signals in the ECG heartbeat examples.
- Transparent lines show test signals close to the corresponding prototypes.