PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Explaining Query Expansion Performance in Information Retrieval

Sourav Saha, Aditya Dutta, Soumajit Pramanik, Mandar Mitra

arXiv:2610.09724v1cs.IRcs.CL

TL;DR

Query expansion addresses vocabulary mismatch, but its effectiveness varies across queries. This paper presents a post hoc framework using ideal expanded queries to analyze that variation; effectiveness consistently decreases as an expanded query moves away from the ideal.

  • Problem

    In unsupervised settings, query expansion can degrade retrieval performance for some queries, and methods with similar average effectiveness may differ across individual queries.

    Source: §1 · pp. 1–2

  • Method

    The paper presents a post hoc framework that empirically approximates ideal expanded queries using complete relevance assessments.

    Source: §3.1 · p. 5, §7 · p. 19

  • Results

    Retrieval effectiveness consistently decreases as an expanded query moves away from the ideal query.

    Source: §7 · p. 19

  • Takeaways & Limitations

    Similarity to an ideal expanded query provides a post hoc perspective for understanding query-expansion performance.

    Source: §7 · p. 19

    Constructing ideal queries in the current framework relies on complete relevance assessments.

    Source: §7 · p. 22

Abstract

from arXiv · show

Query Expansion (QE) techniques have long been widely used in Information Retrieval (IR) to address the vocabulary mismatch problem. They remain relevant in modern retrieval systems, including those based on large language models (LLMs). However, no single QE method consistently outperforms others across all queries. This work seeks to explain the variation in QE performance through two complementary perspectives. The first is the concept of an Ideal Expanded Query (IEQ)--a hypothetical query that maximizes retrieval effectiveness with a downstream BM25 retrieval model. The second is a separability perspective, which quantifies how distinctly relevant and non-relevant documents are scored for a given expanded query using Cohen's (d). We develop a separability measure and practical formulations to approximate the IEQ and investigate how these factors relate to retrieval effectiveness. Extensive experiments on the TREC Robust collection, TREC DL 2019-2022 passage collections, and TREC DL 2019-2020 document collections reveal several interesting patterns. In particular, we find that expanded queries that are closer to the ideal expanded query tend to achieve higher retrieval effectiveness. We further show that the separability of relevant and non-relevant documents provides a complementary perspective for understanding QE performance.

1 Introduction

Query expansion addresses vocabulary mismatch, but its effects vary across queries and methods. This section introduces two perspectives for explaining that variation: ideal expanded queries and document-score separability.

1 Introduction

  • In unsupervised settings, expansion can degrade performance for some queries, and methods with similar average effectiveness can behave differently query by query.The paper frames this variation as a question of what makes one expanded query more effective than another.Source: §1 · p. 2
  • The post hoc framework aims to explain differences across queries and QE methods with interpretable properties, including whether a query is generally hard for automatic expansion and which terms were missed or weighted poorly.These diagnoses are intended as a concrete starting point for exploring possible improvements.Source: §1 · pp. 2–3
  • An ideal expanded query jointly represents good expansion terms and weights as a query vector that would yield perfect or near-perfect Average Precision for a retrieval model.The study examines how expanded queries resemble this ideal and how similarity relates to observed retrieval performance.Source: §1 · p. 2
  • The study also treats retrieval as discrimination between relevant and non-relevant documents, testing whether stronger separation between their BM25 scores is associated with better effectiveness.The authors empirically investigate the strength of this association.Source: §1 · p. 2
  • Experiments cover the Robust collection, TREC DL 2019–2020 document collections, and TREC DL 2019–2022 passage collections.The introduction says the results address the stated research questions but does not specify their outcomes in these passages.Source: §1 · p. 3

2 Related work

Related work spans query-expansion methods, explainable retrieval, and their intersection. The paper positions its contribution as local, developer-oriented analysis of why expansion behaves differently for particular queries.

2.1 Query expansion

  • Earlier query-expansion research largely sought better effectiveness through term selection and weighting, while systematic study of effectiveness on individual queries received little attention.The review covers traditional methods such as Rocchio, PRF, RM1, RM3, and Bo1, alongside neural and LLM-based approaches.Source: §2 · p. 3, §2.1 · p. 3
  • The review notes that modern LLM-generated expansion content is often weighted heuristically, although classical schemes such as Rocchio and RM3 have also been used for LLM-based expanded queries.The cited examples include concatenating generated content with the original query or repeating query terms before concatenation.Source: §2.1 · p. 4

2.2 Explainable IR

  • Explainable IR includes post-hoc explanations and interpretable-by-design models, with explanations aimed either at end users or at system developers.The paper places its work in the developer-oriented category, aiming to help researchers diagnose, compare, and improve QE techniques.Source: §2.2 · p. 4

2.3 Query Expansion applied to Explainable IR and vice versa

  • Prior work applied explainability ideas to QE globally, whereas this paper seeks local explanations of why a QE method performs well or poorly for a specific query.The review also describes approaches that use expanded queries to approximate neural retrieval behavior and assess fidelity through ranking correlation.Source: §2.3 · pp. 4–5

3 Proposed framework

The framework explains query-expansion performance by constructing an ideal expanded query, comparing expanded queries with it, and measuring how they separate relevant from non-relevant documents.

3 Proposed framework

  • The framework has three components: construct QIEQ, measure an expanded query’s similarity to it, and quantify separation between relevant and non-relevant documents.Source: §3 · p. 5

3.1 Construction of QIEQ

  • QIEQ is a conceptual target approximated using complete relevance information, through DFO optimization of AP or a least-squares surrogate for high relevant-document scores.Source: §3.1 · p. 5

3.1.1 Dynamic Feedback Optimization (DFO) [6].

  • DFO starts from a Rocchio query and iteratively adjusts expansion-term weights to maximize AP, retaining updates only when AP improves.Source: §3.1.1 · p. 6

3.1.2 Least-Squares Surrogate.

  • The least-squares surrogate seeks a differentiable objective that separates relevant from non-relevant documents because AP lacks a convenient differentiable form.Source: §3.1.2 · p. 6

3.2 Relation between an EQ’s similarity to QIEQ and its retrieval effectiveness

  • The study measures cosine similarity between an expanded query and QIEQ, then correlates similarity with AP across query variants and averages correlations across queries.Source: §3.2 · p. 7

3.3 How well can an EQ separate relevant and non-relevant documents?

  • The second perspective measures how well an expanded query separates relevant and non-relevant documents using their BM25 scores.Source: §3.3 · p. 7

3.3.1 Separability of BM25 scores.

  • Cohen’s d quantifies separation between relevant and non-relevant BM25 score distributions while accounting for score variability.Source: §3.3.1 · p. 7
  • The analysis correlates each variant’s separability score with AP, then averages those correlations across queries.Source: §3.3.1 · p. 8

4 Experimental setup

The experiments evaluate five query-expansion approaches across TREC and MS MARCO collections, using BM25 retrieval and relevance-judgment-based ideal-query approximations.

4.1 Datasets

  • The setup uses TREC Robust and MS MARCO passage and document collections evaluated with TREC Deep Learning queries from 2019 through 2022.Source: §4.1 · p. 8

4.2 QE approaches explored

  • Five QE approaches cover classical statistical methods, contextualized CEQE, and two HyDE variants, with 90 expanded-query variants considered per query.Source: §4.2 · p. 8
  • Most methods vary pseudo-relevant-document count over {10, 20, 30, 40} and expansion-term count over {15, 25, 35, 45, 55}.Source: §4.2 · p. 8

4.3.1 Retrieval setup.

  • BM25 is the downstream retrieval model, using k1 = 1.2 and b = 0.75, with Porter stemming during indexing and for neural expansion terms.Source: §4.3.1 · p. 9

4.3.2 Ideal Expanded Query Construction with DFO.

  • DFO’s computational cost limits its starting query to the top 200 terms and ends iterative search when AP reaches or crosses 0.9.Source: §4.3.2 · p. 9
  • The least-squares ideal query uses judged documents, gradient descent for 5,000 epochs, cosine annealing, and two regularization values.Source: §4.3.2 · p. 9

5 Results

This section examines how expanded-query similarity to an ideal query and separability between relevant and non-relevant document scores relate to retrieval effectiveness. Greater similarity and separability are associated with higher AP, though these relationships are imperfect and vary across collections.

5.1 Retrieval effectiveness of QIEQ

  • DFO- and LS-generated ideal queries both achieve high MAP, with broadly similar performance except on DL21-22 Passage, where DFO performs better.The best scores on Robust04 and DL 19-20 Passage are described as near-perfect.Source: §5.1 · p. 10, Table 2 · p. 10
  • Different ideal-query constructions can yield near-perfect retrieval despite moderate term overlap, so the ideal expanded query is a family of effective representations rather than one unique query.DFO–LS term overlap is higher on DL21-22 Passage, at 65%–75%.Source: §5.1 · p. 11

5.2 Correlation between Sim(Qexp, QIEQ) and AP(Qexp)

  • Synthetic expanded queries show consistently high correlations between similarity to the ideal query and AP, supporting proximity as a strong predictor of retrieval effectiveness.The analysis rotates an ideal query vector in controlled steps to generate a broad range of synthetic effectiveness values.Source: §5.2 · p. 12

5.3 Correlation for ‘real’ expanded queries

  • Real expanded queries show positive, generally moderate-or-higher correlations between proximity to ideal queries and AP, though weaker than in the synthetic analysis.Correlations are at least 0.40; LS-based ideals correlate more strongly on all collections except DL21-22 Passage, where DFO does better.Source: §5.3 · p. 13

5.4 Correlation between separability and performance

  • The separability analysis relates Cohen’s d for BM25 scores of judged relevant and non-relevant documents to expanded-query AP; its results are not included in this part.Source: §5.4 · p. 13
  • Separability’s correlation with AP is generally stronger than its correlation with real expanded queries’ retrieval effectiveness, reaching ρ=0.7849 on Robust.Source: §5.4 · p. 14
  • Better separation of relevant and non-relevant document scores is associated with good retrieval effectiveness, but high AP does not guarantee high separability.Some expanded queries achieve good AP despite relatively low score separation.Source: §5.4 · p. 14

5.5 Explaining poor performance of different queries

  • Correlations for QLS are slightly lower, and correlations for the DL21-22 Passage collection are relatively weak.Source: §5.5 · p. 15
  • Similarity between expanded queries and the ideal query is positively associated with aggregate AP; QLS shows moderate-to-strong correlations across all collections, and variants closer to ideal tend to achieve higher retrieval effectiveness.Source: §5.5 · p. 15, Table 7 · p. 15

6 Discussion: explaining variations in the similarity–effectiveness relationship

The discussion validates a judged-document proxy for AP, then examines query trajectories and angular spread to explain variation in similarity–effectiveness relationships.

6.1 AP vs. Restricted AP of Qexp

  • Restricted AP ranks only explicitly judged documents, and its consistently high Pearson and Spearman correlations with collection-wide AP support its use as a proxy.The authors report that this evaluation also substantially reduces computational cost.Source: §6.1 · p. 16, Table 8 · p. 16

6.2 Restricted AP vs. Similarity to QIEQ

  • For each real expanded query, the analysis rotates the ideal-query vector through a full circle in the plane it shares with that query, sampling 500 synthetic queries.Restricted AP and similarity to the ideal query are computed at each sampled point.Source: §6.2 · pp. 16–17

Observations.

  • Synthetic queries more similar to the ideal query consistently achieve higher Restricted AP across the examined collections.The reported pattern also matches the strong correlations found in the earlier synthetic-query analysis.Source: Observations · p. 17
  • Across individual trajectories, similarity and effectiveness are positively related, but differences between trajectories weaken the correlation across real expanded queries.This accounts for the weaker correlations reported for real expanded queries on the DL19-20 Document collection.Source: Observations · p. 17

6 Discussion: explaining variations in the similarity–effectiveness relationship

  • Computing collection-wide AP for synthetic queries is expensive, and truncating them to positive-weight terms excludes negative-weight terms from similarity calculations.The authors note that these omitted terms can occur preferentially in non-relevant documents and be detrimental to effectiveness.Source: §6 · p. 16, Observations · p. 17

6.3 Angular spread of expanded versions of a query

  • Real expanded queries generally occupy a narrow angular range while their AP varies more; the resulting correlations can be sensitive to small numerical differences.The authors suggest exploring more discriminatory measures for distinguishing expanded queries.Source: §6.3 · pp. 18–19

7 Limitations and future work

The framework uses an ideal expanded query to analyze QE effectiveness, and reports that proximity tracks retrieval performance while score separability offers a complementary perspective. The discussion identifies non-unique ideal regions, similarity-measure sensitivity, and reliance on complete relevance assessments as limitations or future-work directions.

7 Limitations and future work

  • Similarity to an ideal query is positively correlated with average precision, and synthetic queries show effectiveness consistently falls as they move farther from the ideal.Source: §7 · p. 19
  • Effective expanded queries separate relevant from non-relevant documents, while proximity to the ideal query helps explain automatic QE failures on some topics.Source: §7 · p. 19
  • The ideal query serves as a reference for identifying useful terms a QE method misses and terms that may lead to query drift.Source: §7 · p. 22
  • An ideal expanded query need not be unique: multiple well-separated expanded queries may each deliver near-perfect retrieval.Source: §7 · p. 22
  • Cosine similarity values span a narrow range relative to AP, so small random variations can substantially reduce their correlation.Source: §7 · p. 22
  • Because the framework constructs ideal queries from complete relevance assessments, learning their representations from training data is proposed as future work.Source: §7 · p. 22

To show that the AP obtained by QLS

The appendix explains why scaling the least-squares ideal query does not change its AP under the stated setup. The resulting ideal queries are normalized to unit vectors.

To show that the AP obtained by QLS

B AP of IEQ vs. number of relevant documents

This appendix examines how the number of relevant documents and the LS regularization parameter relate to ideal-query AP and overlap. It reports a negative AP relationship with relevance-set size and favorable overlap under moderate regularization.

B AP of IEQ vs. number of relevant documents

D Variability across queries

The section compares variability across queries with variability among expansion methods for the same query. Across the examined collections and ideal-query representations, between-query variation is substantially greater.

D Variability across queries

Loading 2610.09724v1…