EXS: Explainable Search Using Local Model Agnostic Interpretability

1 INTRODUCTION

• Neural retrieval models risk becoming functional black boxes as deep learning advances toward broader deployment.Interpretability matters to researchers and developers seeking to debug or improve models, and to users and experts seeking explanations or audits.Source: §1 · p. 1
• Tools for explaining black-box models have emerged in other tasks, but little work has addressed ranking outputs.The introduction notes that artifacts, data-specific biases, and intents can be hard to spot with quantitative measures.Source: §1 · p. 1
• Ranking explanations must address query intent, why a document is relevant, and why it outranks another document.These views can help diagnose learned intent and isolate errors associated with training-data biases or modeling and training artifacts.Source: §1 · p. 1
• EXS is designed to help end users understand text-based neural retrieval models through these ranking-related questions.The paper describes EXS as a system for helping users better understand such models.Source: §1 · p. 1

2 INTERPRETABILITY IN MACHINE LEARNING

• Model-introspective approaches rely on models whose components can be meaningfully inspected, such as a decision-tree path or a linear-model feature weight.Examples include decision trees, rules, additive models, and attention-based networks.Source: §2 · pp. 1–2
• Model-agnostic methods treat the original model as a black box, producing explanations from its outputs, perturbed inputs, or both.Local interpretation explains a single decision; global interpretation approximates overall behavior with a simpler model.Source: §2 · p. 2
• EXS adapts LIME, a local model-agnostic method for classifiers, to explain pointwise rankers that score query-document pairs.The paper notes that little work had addressed explanations for a black-box retrieval model’s result list, despite prior work on global ranking interpretability.Source: §2 · p. 2

INTERPRETABILITY FOR RANKING MODELS

• LIME approximates a classifier locally by training a simple linear SVM on word features from randomly perturbed documents.The perturbations remove random words, and the model is trained to match the black-box classifier’s outputs within a defined locality.Source: INTERPRETABILITY FOR RANKING MODELS · p. 2
• For ranking, EXS converts a ranker’s output for a query-document pair into a relevant/irrelevant class distribution so LIME can label perturbed documents.The paper considers top-k binary, score-based, and rank-based conversions; top-k binary assigns relevance probability 1 when the score exceeds the kth document’s score.Source: INTERPRETABILITY FOR RANKING MODELS · p. 3
• The top-k binary conversion assumes a perturbed document is relevant when its score exceeds the score of the kth document in the retrieved list.The probability of the irrelevant class is defined as one minus the relevance probability.Source: INTERPRETABILITY FOR RANKING MODELS · p. 3
• The local explanation model uses words as features, with coefficient signs and magnitudes indicating which document words support relevance.The training examples are generated by perturbing the document within the chosen locality and querying the black-box model.Source: INTERPRETABILITY FOR RANKING MODELS · pp. 2–3

4 EXS – AN EXPLAINABLE SEARCH SYSTEM

• EXS lets users select a retrieval model, corpus, rank depth, and probability conversion while viewing search results beside an explanation panel.The interface also provides an Explain Intent control.Source: §4 · p. 3

4.1 Visualizing Explanations

• For a single document, EXS plots word coefficients as bars; for a selected pair, it explains which words distinguish the higher-ranked document relative to the comparison document.The interface uses green bars for the relevant class, and the pairwise explanation sets its threshold using the second document’s rank.Source: §4.1 · p. 3
• For query intent, EXS sums word coefficients across top-k documents and displays the highest aggregated words and coefficients.The user can trigger this view from the Explain Intent button.Source: §4.1 · p. 3, Figure 2 · p. 3

4.2 Architecture and Implementation Details

• The client-side EXS application uses a REST API and Lucene to retrieve documents from four TREC news collections.The collections are AP, LATIMES, Robust04, and Financial Times; the implemented neural models include DRMM and DESM.Source: §4.2 · p. 3
• EXS generates explanations with a modified LIME implementation using 2,000 perturbed samples, while users control the number of words shown.The other primary LIME parameter is the number of words in the explanation.Source: §4.2 · p. 3

• The DRMM examples use Robust04 and 300-dimensional GloVe embeddings trained on the same collection to create input histograms.Source: §5 · pp. 3–4
• For the “Rail Strikes” query using DRMM, Figure 3 compares AP890710-0178 with AP890713-0045; its one-sided explanation shows railway and union as difference makers favoring AP890710-0178.Source: Figure 3 · p. 4, §5 · p. 4
• DRMM’s query-intent explanation identifies transport and union as relevant indicators, and generic terms such as also and for as irrelevant indicators.The paper reports that DRMM captures the query’s correct intent.Source: §5 · p. 4
• For AP890710-0178, unrest and walkouts indicate relevance, while jimmy indicates the contrary; removing strike could be highly detrimental.The explanation suggests removing jimmy and state can increase the document’s relevance.Source: §5 · p. 4

• EXS adapts posthoc, model-agnostic LIME for rankers, converting scores into relevant/irrelevant probabilities through Top-k binary, score-based, and rank-based approaches.Source: §6 · p. 4
• Three anecdotal DRMM scenarios demonstrate LIME explanations for ranking questions, and the authors describe EXS as useful for diagnosing and auditing ad hoc retrieval models.Source: §6 · p. 4
• Future work includes diversity-based and multimodal retrieval, and investigating whether diagnostics affect search behavior in medical and legal applications.Source: §6 · p. 4