Source-linked AI summary
Towards Axiomatic Explanations for Neural Ranking Models
Michael Völske, Alexander Bondarenko, Maik Fröbe, Matthias Hagen, Benno Stein, Jaspreet Singh, Avishek Anand
TL;DR
Neural ranking models are difficult to interpret, and little is known about how their decisions correspond to established IR principles. This paper reconstructs rankings from combinations of retrieval axioms; the axioms explain large score differences better than close-result fluctuations, while strict preconditions limit their applicability.
Problem
Neural ranking models can achieve strong retrieval effectiveness, but their complexity makes them difficult to interpret and their correspondence with established IR principles is poorly understood.
Source: §1 · p. 1Method
Results
Axiomatic explanations work reliably for pairs with very different retrieval scores, while pairs with more similar scores are harder to explain.
Source: §6 · p. 9Takeaways & Limitations
The current axiom set does not fully capture ranking criteria, and additional axioms or less restrictive preconditions may broaden its explanatory coverage.
Source: §5.4 · p. 8Strict axiom preconditions limit applicability; on Robust04, term-frequency axiom preconditions hold for only about 7% of document pairs.
Source: §5.4 · p. 8
Abstract
from arXiv · showhide
Recently, neural networks have been successfully employed to improve upon state-of-the-art performance in ad-hoc retrieval tasks via machine-learned ranking functions. While neural retrieval models grow in complexity and impact, little is understood about their correspondence with well-studied IR principles. Recent work on interpretability in machine learning has provided tools and techniques to understand neural models in general, yet there has been little progress towards explaining ranking models. We investigate whether one can explain the behavior of neural ranking models in terms of their congruence with well understood principles of document ranking by using established theories from axiomatic IR. Axiomatic analysis of information retrieval models has formalized a set of constraints on ranking decisions that reasonable retrieval models should fulfill. We operationalize this axiomatic thinking to reproduce rankings based on combinations of elementary constraints. This allows us to investigate to what extent the ranking decisions of neural rankers can be explained in terms of retrieval axioms, and which axioms apply in which situations. Our experimental study considers a comprehensive set of axioms over several representative neural rankers. While the existing axioms can already explain the particularly confident ranking decisions rather well, future work should extend the axiom set to also cover the other still "unexplainable" neural IR rank decisions.
1 INTRODUCTION
Neural ranking models achieve strong retrieval effectiveness but are difficult to interpret, motivating a test of whether established IR axioms can explain their decisions. The paper combines axioms to reconstruct neural rankings and finds that existing axioms explain confident score differences better than close-result fluctuations.
1 INTRODUCTION
- Machine-learned ranking models combine relevance features based on training data, but their complexity makes their decisions difficult to interpret.The passage notes that some reported performance gains have been called into question [28].Source: §1 · p. 1
- The framework turns axioms into pairwise document preferences and trains an explanation model to reproduce a ranker's ordering from those preferences.Its parameters indicate which constraints matter to a ranker, while reconstruction fidelity can expose blind spots in the axiom set.Source: §1 · p. 1
- The study combines multiple retrieval axioms to reconstruct neural rankings, extending prior work that used axioms to explain or improve ranking decisions.The authors describe this as the first extensive experimental study specifically combining many axioms to reconstruct neural rankings.Source: §1 · pp. 1–2
- The study also explores whether axiomatic explanations vary across queries or across positions in result rankings.Source: §1 · p. 2
- Large retrieval-score differences are reasonably well explained by existing axioms, while small fluctuations among close results are much less so.The authors identify additional axioms and broader applicability of existing ones as future directions.Source: §1 · p. 2
2 RELATED WORK
Related work develops axioms to characterize retrieval behavior and applies them to improve, diagnose, or explain ranking systems. This paper combines those axioms to reconstruct rankings, complementing prior approaches to neural diagnostics and model interpretability.
2 RELATED WORK
- Axiomatic IR seeks to explain a “good” retrieval function through mathematically described formal constraints, unlike commonly used scoring functions such as BM25, which quantify a specific, narrow notion of relevance.Source: §2 · p. 2
- Axioms span relevance aspects including term frequency, document length, query aspects, semantic similarity, and term proximity; most prior studies considered axioms individually.Other proposed groups include evaluation, link graphs, learned ranking functions, and multi-criteria relevance.Source: §2 · p. 2
- Earlier work combined 23 axioms to rerank top-k results and improve basic retrieval models; this study instead reconstructs a model's rankings from weighted axiom preferences.Source: §2 · p. 2
- Prior neural-IR diagnostics tested individual axioms or controlled manipulations, whereas this study combines 20 axioms on realistic TREC-style queries and collections.The passage describes this reconstruction approach as complementary to controlled and synthetic diagnostic settings.Source: §2 · p. 2
- Axiomatic explanations differ from post-hoc feature attribution by using axioms as candidate explanations and a learning framework to measure fidelity [18] [47] [48].Source: §2 · p. 3
3 AXIOMATICALLY EXPLAINING RANKINGS
The framework explains a retrieval model's rankings by reconstructing pairwise orderings from axiomatic preferences. It adapts retrieval axioms to document-pair ranking and uses an inspectable explanation model to reveal how those preferences contribute.
3 AXIOMATICALLY EXPLAINING RANKINGS
- For each document pair, the framework computes an axiom-preference vector and trains an explanation model to reproduce the original ranking's pairwise ordering.The model's parameters indicate how the axioms contribute to the ranking under scrutiny.Source: §3 · p. 3, Figure 1 · p. 4
3.1 Operationalizing Retrieval Axioms
- Axioms are operationalized as ternary preferences—favor the first document, favor the second, or express no preference—and only axioms restatable as pairwise ranking preferences are included.The authors adapt the operationalization of Hagen et al..Source: §3.1 · p. 3
- For term-frequency axioms, the authors relax equality conditions, strengthen inequality conditions, and generalize some constraints to multi-term queries.For example, TFC1 allows document lengths within 10% and requires at least a 10% term-frequency difference; TFC3 and TDC are also generalized.Source: §3.1 · p. 3
- The operationalized set covers constraints on term frequency, document length, lower bounds, query aspects, semantic similarity, and term proximity.Examples include preferring shorter documents at equal query-term frequency, documents containing every query term, and documents with more occurrences of less-similar query terms.Source: §3.1 · p. 3
3.2 Aggregating Axiom Preferences
- The explanation model learns binary pairwise decisions from axiom preferences, balancing reconstruction fidelity against interpretability because higher-capacity models tend to be less inspectable.The study uses a random forest for useful but limited feature-importance inspection; simpler models may be more interpretable once the axiom set is more comprehensive.Source: §3.2 · pp. 3–4
4 EXPERIMENTAL SETUP
The experiments test how well axioms reconstruct ranking decisions across two standard collections, three classical and five neural retrieval models, and 200 queries. Pairwise preferences from the rankers and axioms train random-forest explanation models, with experiments also varying the scope at which explanations are fit.
4 EXPERIMENTAL SETUP
- The study compares axiomatic explanations across two standard datasets, 200 queries, and three classical and five neural ranking models.It examines the top 1,000 results and fits random-forest explanation models to sampled document pairs.Source: §4 · p. 4
4.1 Collections
- Robust04 experiments use 100 held-out topics; MS MARCO experiments use 100 of the official test queries and document labels derived from passage judgments.The collections differ in query style and document characteristics; MS MARCO labels assume a document with a relevant passage is relevant, which may introduce label noise.Source: §4.1 · p. 4
4.2 Ranking Models
- The study uses three neural rankers on Robust04 and two BERT-based rankers on MS MARCO, alongside BM25, TF-IDF, and PL2 for comparison.The neural models are selected for their respective settings; studying all rankers on both datasets is left for future work.Source: §4.2 · pp. 4–5
4.3 Explanation Parameters
- A random forest predicts each retrieval model’s pairwise ranking preference from the preferences of 20 axioms, using ten-fold cross-validation.Mirrored document pairs stay in the same fold to avoid train-test information leakage; the experiment samples all pairs among the top 20 ranks and 1% of remaining pairs.Source: §4.3 · p. 5
4.4 Explanation Model Locality
- Explanation models are fit at full-ranking or subset scopes, varying locality by query, ranking position, and score difference.Ranking position and score difference are distinguished because documents can occupy different positions despite receiving the same score; the study uses 24 bins for position and score differences.Source: §4.4 · p. 5
5 RESULTS
Axiomatic explanations reproduce rankings more successfully for large score differences than for close document pairs, with fidelity varying across collections and rankers. The results also show that current axioms leave important ranking behavior unexplained, particularly in neural models.
5.1 Overall Explanation Fidelity
- 0.54 or higher fidelity is achieved across examined models and settings; PL2 approaches 80% on Robust04, while MS MARCO neural results are poorer and not directly comparable across collections.Neural rankers on Robust04 fall within the fidelity range of TF-IDF and BM25.Source: Table 2 · p. 6, §5.1 · p. 6
- Query-scoped explanations are the most straightforward and usually best-performing setup; shuffled-ranking fidelity stays below 0.52, while studied rankings exceed chance.More complicated explanation scopes do not strongly influence fidelity overall.Source: §5.1 · p. 6
5.2 Explanation Fidelity by Score Difference
- Explanation fidelity rises with score difference, especially on Robust04; low-difference pairs are poorly explained on both datasets, and distant pairs largely account for Robust04's advantage.The figure reports per-query fidelity by score-difference bin, with 95% confidence intervals.Source: Figure 2 · p. 7, §5.2 · p. 7
5.3 Ranking Quality and Explanation Fidelity
- Spearman correlations between nDCG and fidelity are 0.16 for DRMM, 0.26 for MP-COS, and 0.2 for PACRR-DRMM, indicating weak but positive association across Robust04 queries.Query 367 is an exception: no system ranks it particularly well by nDCG, yet fidelity is high across models.Source: §5.3 · p. 7
5.2 Explanation Fidelity by Score Difference
- REG and PROX4 recur across models and most score-difference bins, while DIV helps explain most models; PL2 uniquely features STMC1 among its top axioms at large differences.PL2 uses exact matches, while STMC1 rewards semantically similar query terms; the authors suggest their document properties may correlate.Source: §5.2 · p. 7, Table 3 · p. 8
5.3 Ranking Quality and Explanation Fidelity
- For query 367 (“piracy”), agreement between axioms and top-10 ranks is rather poor, though overall good explanation fidelity comes from more distant pairs; query 425 (“counterfeiting money”) has better top-10 agreement, led by term-proximity axioms.For query 367, DRMM’s ranking is best explained by REG and DIV axioms.Source: §5.3 · p. 8
5.4 Limitations and Summary
- Adding two ranker-specific classical-model axioms raises classical fidelity above 99%, but improves neural fidelity only moderately, leaving neural ranking behavior less explained.The authors hypothesize that current axiom formulations lack reliable indicators for some basic relevance signals.Source: §5.4 · p. 8
- Several axioms have restrictive preconditions, limiting coverage; on Robust04, term-frequency preconditions hold for about 7% of pairs, versus 21% for PROX1–3 and 36% for LB1.The authors suggest the 10% relaxation of equality constraints may still be too strict.Source: §5.4 · p. 8
6 CONCLUSION & FUTURE WORK
The paper introduces a framework for explaining retrieval rankings through adherence to axiomatic constraints. Across Robust04 and MS MARCO, axioms explain high-confidence document pairs more reliably than close-score pairs, while important search aspects remain uncovered.
6 CONCLUSION & FUTURE WORK
- The framework explains rankings by measuring how well a system’s decisions adhere to a set of axiomatic constraints.The experiment uses 20 literature axioms and a random forest to reconstruct pairwise orderings on Robust04 and MS MARCO.Source: §6 · pp. 8–9
- Axiomatic explanations work reliably for document pairs with very different retrieval scores, but pairs with similar scores are harder to explain.The study covers eight retrieval systems, including five neural rankers and three classical scoring functions.Source: §6 · p. 9
- Explanation fidelity is higher on Robust04 than on MS MARCO, with query and document differences proposed as possible explanations.The paper says further investigation into the discrepancy is warranted.Source: §6 · p. 9
- Known axioms omit aspects relevant to modern search, including timeliness, readability, and user preferences expressed through prior interactions.Formalizing these aspects was beyond the paper’s scope.Source: §6 · p. 9
- The authors propose adding well-grounded axioms for uncovered retrieval aspects to improve explanations.They describe the framework as a potential complement to controlled studies for helping system designers or end users interpret concrete rankings.Source: §6 · p. 9