Source-linked AI summary
A study on the Interpretability of Neural Retrieval Models using DeepSHAP
Zeon Trevor Fernando, Jaspreet Singh, Avishek Anand
TL;DR
Neural retrieval models can rank documents well, but it remains difficult to explain why a document is relevant, and adapting DeepSHAP to this setting requires choosing a suitable document reference. The paper evaluates reference-document constructions and compares DeepSHAP with LIME, finding reference-sensitive explanations and substantial differences between the methods.
Problem
Neural retrieval models can rank documents effectively, but their relevance decisions are difficult to explain, and explanations for complex neural ranking models have received little attention.
Source: §1 · p. 1Method
The paper applies DeepSHAP to three neural retrieval models, exploring document reference inputs while comparing its explanation terms with LIME.
Source: §1 · p. 1Results
Takeaways & Limitations
The study highlights unresolved problems in model-introspective interpretability for neural retrieval models and points to further work on approaches that account for discrete inputs.
Source: §6 · p. 4The reference-document variants assume that their words are irrelevant to the query, an assumption that guides their construction.
Source: §3 · p. 2
Abstract
from arXiv · showhide
A recent trend in IR has been the usage of neural networks to learn retrieval models for text based adhoc search. While various approaches and architectures have yielded significantly better performance than traditional retrieval models such as BM25, it is still difficult to understand exactly why a document is relevant to a query. In the ML community several approaches for explaining decisions made by deep neural networks have been proposed -- including DeepSHAP which modifies the DeepLift algorithm to estimate the relative importance (shapley values) of input features for a given decision by comparing the activations in the network for a given image against the activations caused by a reference input. In image classification, the reference input tends to be a plain black image. While DeepSHAP has been well studied for image classification tasks, it remains to be seen how we can adapt it to explain the output of Neural Retrieval Models (NRMs). In particular, what is a good "black" image in the context of IR? In this paper we explored various reference input document construction techniques. Additionally, we compared the explanations generated by DeepSHAP to LIME (a model agnostic approach) and found that the explanations differ considerably. Our study raises concerns regarding the robustness and accuracy of explanations produced for NRMs. With this paper we aim to shed light on interesting problems surrounding interpretability in NRMs and highlight areas of future work.
1 INTRODUCTION
Neural retrieval models can rank documents effectively, but their relevance decisions are difficult to explain, and interpretability work for neural ranking has been scarce. This paper studies DeepSHAP for neural retrieval models and finds that its explanations depend strongly on the reference input and differ considerably from LIME.
1 INTRODUCTION
- Little work had explained complex neural ranking outputs, despite the importance of understanding search decisions and the promise of neural retrieval models.Explanations can help debug models, identify training-data bias, and support trust.Source: §1 · p. 1
- DeepSHAP adapts DeepLift to estimate Shapley-value attributions for input features, measuring their contribution to a prediction relative to an average prediction.The method produces feature attributions that can be visualized over input words or pixels.Source: §1 · p. 1
- DeepSHAP explanations for neural retrieval models depend strongly on the reference input, while their explanations differ considerably from LIME’s.The authors ask what document can serve as an IR equivalent of the plain black image used as a reference in computer vision.Source: §1 · p. 1
2 RELATED WORK
Interpretability work includes model-agnostic and model-introspective approaches, alongside diagnostic studies of neural IR. The paper identifies its contribution as the first study of model-introspective interpretability specifically for neural retrieval models.
2 RELATED WORK
- Interpretability methods include model-agnostic explanations that treat a model as a black box and model-introspective approaches that inspect components or generate attributions.The latter include interpretable model structures and gradient-based methods.Source: §2 · p. 2
- Prior neural IR work includes diagnostic analyses, LIME-based explanations for document relevance, and methods that approximate query intent encoded by neural ranking models.The diagnostic work uses retrieval axioms for manual error analysis or diagnostic-dataset construction [9]; EXS adapts LIME [12] [18], while [17] approximates rankings with an expanded query.Source: §2 · p. 2
- The authors identify this as the first work on model-introspective interpretability specifically for neural retrieval models.Source: §2 · p. 2
3 DEEPSHAP FOR IR
The paper adapts DeepSHAP to explain document relevance by attributing predictions to document words relative to a reference input. It fixes the query reference to the query being explained and explores multiple document-reference constructions.
3 DEEPSHAP FOR IR
- DeepSHAP uses DeepLIFT to approximate Shapley values by comparing an input and its prediction with a reference input and reference output.Feature-importance scores are computed using derivatives and propagated through the network with the chain rule.Source: §3 · p. 2
- For ranking, the query reference is fixed to the query being explained, while document references vary to estimate an average output near that query.The document words’ Shapley values indicate their importance to the model’s relevance prediction.Source: §3 · p. 2
- The explored reference documents include OOV tokens, low-IDF words, and low-query-likelihood words intended to be generally irrelevant to the query.For DRMM and MatchPyramid, OOV embeddings are all zeros; PACRR-DRMM uses the vocabulary’s average embedding for OOV.Source: §3 · p. 2
- Other document references are sampled from the collection outside the query’s top 1,000 results or from the bottom of those retrieved results.These are respectively the collection-random and top-k-list-random reference variants.Source: §3 · p. 2
4 EXPERIMENTAL SETUP
The experiments ask whether DeepSHAP explanations vary with the document reference and whether any reference yields more accurate local explanations. They use three neural retrieval models, Robust04 topics, and LIME explanations as a proxy for evaluation.
4 EXPERIMENTAL SETUP
- The experiments test DeepSHAP’s sensitivity to reference inputs and whether one reference produces the most accurate local explanation.Source: §4 · p. 2
4.1 Neural Retrieval Models
- The evaluated models are DRMM, MatchPyramid, and PACRR-DRMM, which use different query-document matching representations and neural architectures.DRMM uses a matching-count histogram; MatchPyramid and PACRR-DRMM use interaction matrices, with PACRR-DRMM applying multiple CNN kernel sizes and k-max pooling.Source: §4.1 · p. 2
4.2 Evaluation
- Experiments use Robust04’s 50 hard topics and the top three retrieved documents per query, with LIME as a proxy because ground-truth explanations are unavailable.LIME’s local linear model achieved over 90% accuracy across the considered NRMs; comparisons use selected explanation terms and ignore polarity.Source: §4.2 · p. 3
5 RESULTS AND DISCUSSION
DeepSHAP explanations vary substantially with the reference document, and the best-performing reference method differs by NRM. The comparison with LIME likewise shows model-dependent differences in how well DeepSHAP captures its terms.
5.1 Effect of reference input document
- DeepSHAP explanations have low overlap across reference inputs—below 50% in most cases—and no reference method consistently wins across NRMs.MatchPyramid's highest overlap is for OOV and QL, while PACRR-DRMM's is for OOV and COL.Source: §5.1 · p. 3
5.2 Accuracy of reference input methods
- DeepSHAP's agreement with LIME varies by model: DRMM captures almost all LIME terms, while PACRR-DRMM recall stays below 0.7.The table compares recall for top-50 and top-100 DeepSHAP terms against top-10, top-20, and top-30 LIME terms on ROBUST04 difficult queries.Source: §5.2 · p. 3, Table 1 · p. 4
- QL is the best reference method for DRMM, suggesting query-specific generic terms are a better background than collection-generic IDF terms.Source: §5.2 · p. 3
- For MatchPyramid, TOPK LIST performs worst and OOV performs best; OOV's zero-vector padding differs from DRMM's filtering of OOV terms.The study identifies these preprocessing choices as important when selecting a reference input.Source: §5.2 · p. 3
- QL is also best for PACRR-DRMM, although DeepSHAP still struggles to recover all LIME terms.Source: §5.2 · p. 3
6 CONCLUSION AND FUTURE WORK
The study finds that DeepSHAP explanations for NRMs depend on the reference input, and that the reference method that best matches LIME depends on the model. It calls for further work that accounts for text discreteness and model preprocessing when generating explanations.
- DeepSHAP explanations are sensitive to the reference input, and the best reference method depends on the NRM.Source: §6 · p. 4
- The authors identify new problems for model-introspective interpretability of NRMs.Source: §6 · p. 4
- Future work should investigate explanation approaches that explicitly account for text discreteness and model preprocessing choices.Source: §6 · p. 4