PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Together: Complementary Query Rewriting Under a Strong RAG Baseline

Sara Shanian, Xiaoqin Yi, Pavlo Ruban, Kurt MacDonald

arXiv:2609.05637v1cs.CLcs.IR

TL;DR

The paper asks whether query rewriting adds value beyond a strong RAG baseline and tests six methods under one fixed retrieval pipeline across three datasets. It finds that complementary ensembles substantially improve enterprise retrieval, while confidence-gated routing captures part of that benefit at lower rewriting cost.

  • Problem

    The paper studies whether query rewriting adds value beyond a strong RAG baseline, addressing limited evidence from comparisons built on weak retrievers.

  • Method

    The study compares six methods across three datasets under a fixed BGE retrieval, cross-encoder reranking, and MMR pipeline, using controlled significance testing.

  • Results

    52.98% HIT@10 is reached by a post-hoc ensemble, improving 13.8 points over the dense baseline, while budget-matched controls capture only about 40% of the gain.

  • Takeaways & Limitations

    Query rewriting is best treated as a complementary coverage source applied through cost-aware routing rather than as a standalone replacement for a strong baseline.

  • Takeaways & Limitations

    The study uses post-hoc ensembles and simulated routing; production validation, broader enterprise corpora, and richer answer-quality evaluation remain future needs.

Abstract

from arXiv · show

A popular way to improve Retrieval-Augmented Generation (RAG) is to rewrite the user's question into several variants and search with all of them. We test whether this actually helps once the underlying search is already strong. Under one fixed, competitive pipeline (BGE dense retrieval, cross-encoder reranking, and MMR diversification), we compare four query-rewriting strategies (S1-S4) against two strong LLM baselines (HyDE, Query2Doc) on three datasets (HotpotQA, AmbigNQ, and the 512K-document EnterpriseRAG-Bench) over three seeds with paired-bootstrap significance tests. Our headline result is that rewriting alone is at best competitive with a strong baseline, but combining methods yields outsized gains because different strategies fail on different questions. A post-hoc union of four methods (S1+S3+S4+HyDE) improves HIT@10 over the baseline by +12.5 points on enterprise data (51.70 vs 39.22), and a five-method union reaches 52.98 (+13.8). Budget-matched controls capture only ~40% of this gain, confirming that complementarity, not retrieval budget, is the primary driver. On HotpotQA the union adds +1.6 to +1.8 points (p<0.001), saturating the all-method oracle; on AmbigNQ the same fusion hurts (-2.4 below the best solo, p<0.001), and we analyze when and why. Because rewriting is expensive, we evaluate in simulation a confidence-gated router that runs rewriting only when the baseline's own top-1 score is low. It captures about half of the enterprise full-merge gain (+4.3 HIT@10) while paying rewriting cost on <40% of queries, and automatically declines to rewrite on AmbigNQ. A downstream answer-quality evaluation confirms the router improves F1 by +1.92 (p<0.01) at roughly 40% of the expansion cost. In short: treat query rewriting as a complementary coverage source, applied through cost-aware routing, not as a standalone replacement for a strong baseline.

1 Introduction

The paper tests whether query rewriting remains useful atop a strong RAG retriever rather than a weak baseline. Rewriting is competitive alone, but combining complementary methods produces larger gains because they fail on different questions.

  • Motivation: The study fixes a competitive pipeline with BGE dense retrieval, cross-encoder reranking, and MMR diversification to test rewriting fairly.Prior comparisons often used bare bi-encoders without reranking.
  • Main findings: Rewriting alone is competitive at best against the no-LLM baseline across the evaluated datasets.Multi-query strategies match or modestly beat S1, while HyDE and Query2Doc perform similarly; no single strategy dominates.
  • Main findings: 52.98% HIT@10 is achieved by a post-hoc ensemble, improving 13.8 points over the S1 baseline and approaching the all-method oracle at 55.74.The ensemble is within approximately 2.8 points of the oracle.
  • Main findings: Budget-matched baselines capture only approximately 40% of the ensemble gain, indicating complementarity rather than retrieval depth drives the improvement.The result supports combining methods that cover different failures.
  • Practical implication: A confidence-gated router realizes about half the enterprise full-merge gain at under 40% of rewriting cost and improves downstream answer F1 by 1.92.The router is designed to invoke rewriting selectively rather than replace the baseline.

2 Related Work

Prior work improves RAG through retrieval, reranking, query expansion, decomposition, and LLM-generated documents. This paper builds on that literature by isolating rewriting under a fixed retrieval pipeline.

  • RAG: RAG research has advanced retrieval and evidence use through methods including Dense Passage Retrieval, Fusion-in-Decoder, and Self-RAG.Systematic studies also show that reranking and chunking affect RAG quality.
  • Query rewriting: Query rewriting spans relevance feedback, neural expansion, and decomposition for multi-hop reasoning.These approaches modify or expand the original question to improve retrieval or reasoning.
  • LLM document generation: HyDE retrieves using an embedding of a generated hypothetical answer, while Query2Doc appends a generated pseudo-document to the query.Both are LLM document-generation baselines evaluated in this paper.
  • Recent directions: Recent RAG work also produces diverse rewrites, plans multi-hop logic, or optimizes rewrite choice under a budget.The paper differs by comparing these ideas under one strong fixed pipeline.

3 Reformulation Strategies

The paper compares a no-LLM baseline, three multi-query rewriting strategies, and two document-generation methods. Their outputs are filtered, retrieved, and merged into a shared ranked list.

  • Method families: Six methods form three families: S1 is the no-LLM baseline, S2–S4 generate alternative queries, and HyDE and Query2Doc generate pseudo-documents.Every non-S1 method makes one LLM call per query; S2 and S3 also issue several retrievals.
  • Method families: S1 searches with the original query unchanged and serves as the strong reranked baseline for all comparisons.It requires no LLM call.
  • Multi-query rewriting: S2 generates related sibling questions, S3 restates the question with synonyms, and S4 forms a comparison question.S2 is highest-variance, S3 targets vocabulary mismatch, and S4 uses one variant.
  • Document generation: HyDE writes a hypothetical answer passage and searches with its embedding, whereas Query2Doc appends a generated pseudo-document to the query.Both methods are expensive and serve as strong solo baselines.
  • Merging: Multi-query variants are quality-filtered, retrieved individually, and merged by retaining each document’s maximum similarity before selecting the global top-K.The same score-based merge is used for cross-strategy ensembles.

4 Experimental Setup

The experiments cover three distinct datasets and use one fixed strong retrieval pipeline across methods. Results are averaged over three seeds with standard retrieval metrics and paired-bootstrap significance tests.

  • Datasets: HotpotQA uses 5,000 training examples, AmbigNQ uses 2,002 development queries, and EnterpriseRAG-Bench uses 470 answerable questions.EnterpriseRAG-Bench contains approximately 512K synthetic enterprise documents.
  • Method cost: Only S1 avoids an LLM call; S4 is the cheapest rewriting strategy, while S3 is most amenable to dictionary-based implementation.The reported experiments nevertheless use the LLM version of S3 for fairness.
  • Pipeline: The fixed pipeline uses bge-base-en-v1.5 embeddings, bge-reranker-base, MMR with λ=0.85, sentence chunks of 512/50, score-based mixing, and GPT-4.1 generation.The pipeline is deliberately strong because prior rewriting studies often omit reranking and MMR.
  • Protocol: Evaluation reports Recall@10, NDCG@10, MRR@10, and HIT@10 across seeds 42, 123, and 2024.Significance versus S1 uses a two-sided paired bootstrap with 2,000 iterations.

5 Results

Under a strong retrieval baseline, rewriting alone is only modestly competitive, while combining methods yields large gains when they recover different missed queries. These gains support a confidence-gated router that improves enterprise retrieval and answer quality while limiting expansion cost.

  • 5.1 Alone, Rewriting Is Competitive at Best: Rewriting strategies match or modestly beat S1 across datasets, but no single method dominates.S2–S4, HyDE, and Query2Doc remain competitive without producing a universally best strategy.
  • 5.1 Alone, Rewriting Is Competitive at Best: S2 reaches the highest HotpotQA HIT@10 at 97.65, while S3 is best on Enterprise HIT@10 at 42.91.Query2Doc leads HotpotQA Recall@10 and NDCG@10; S2 is also high-variance and costly, whereas S4 is the cheapest practical rewriting strategy.
  • 5.2 Different Methods Fail on Different Questions: Query2Doc rescues 14.8% of S1 misses on Enterprise, while other methods rescue substantial fractions on AmbigNQ and HotpotQA.These rescue rates show why solo scores can hide complementary successes and failures across queries.
  • 5.2 Different Methods Fail on Different Questions: +12.8 points is the Enterprise oracle-union headroom over the best single method, compared with +1.7 on HotpotQA and +2.7 on AmbigNQ.Enterprise retrieval is more diverse, with mean pairwise Jaccard@10 of 0.14–0.45 versus 0.55–0.99 on AmbigNQ.
  • 5.3 Combining Methods: 52.98 HIT@10 is achieved by the five-method Enterprise ensemble, +13.8 over S1; the four-way S1+S3+S4+HyDE ensemble reaches 51.70, +12.5.HotpotQA fusion reaches 99.40, effectively saturating its 99.37 oracle, whereas every AmbigNQ combination underperforms the best solo S3 by 2.5 points.
  • 5.3 Combining Methods: Budget-matched controls capture only 40–41% of the four-way Enterprise ensemble gain, indicating that complementarity contributes beyond retrieval budget.TopK-100 reranking and multi-pass HNSW both converge to approximately 44.3 HIT@10, versus the ensemble’s +12.5-point gain over S1.
  • 5.4–5.5 Cost-Aware Routing: The confidence-gated router captures +4.33 Enterprise HIT@10 while expanding 39% of queries and improves answer F1 by +1.92.It nearly abstains on AmbigNQ, expands 1.2% of HotpotQA queries, and preserves near-full-merge answer quality at roughly 40% of expansion cost.

6 Discussion

The discussion reframes rewriting as a complementary coverage source rather than a standalone replacement for a strong baseline. Its benefits depend on productive disagreement among methods, while overlap, leakage concerns, and cost shape when it should be used.

  • When combination fails: AmbigNQ fusion hurts because near-identical retrieval lists let shared non-relevant documents crowd out the small relevant set.Jaccard@10 ranges from 0.55–0.99, with S2–S4 near 0.99; the five-way ensemble is significantly below the best solo method.
  • Complementarity: Enterprise gains arise when methods disagree productively, producing low-overlap retrieval and greater headroom for complementary coverage.The paper contrasts this enterprise regime with AmbigNQ’s high-overlap, multi-relevant setting.
  • Complementarity: The best solo strategy changes by dataset, supporting ensemble use rather than reliance on one universally dominant rewriting method.S3 is best on Enterprise HIT@10, Query2Doc on HotpotQA Recall, and S2 on AmbigNQ Recall.
  • Scope and leakage: Leakage cannot be fully ruled out on public web-QA datasets, whereas EnterpriseRAG-Bench post-dates model training cutoffs and shows the largest complementarity gains.The enterprise documents were released in May 2026, after the stated training cutoffs of the retrievers, rerankers, and query-generation model.
  • Practical guidance: A practical policy starts with reranked, MMR-diversified S1, adds Query2Doc by default, and gates rewriting using a reranker-score threshold.The guidance also favors S4 over S2/S3 when adding a cheap rewriting member.

7 Conclusion

Under a strong fixed RAG pipeline, rewriting is only modestly useful alone but becomes substantially more effective when methods cover different failures. Cost-aware routing preserves part of this benefit while limiting expansion expense.

  • Core conclusion: +8.3 to +9.6 points HIT@10: post-hoc enterprise ensembles beat the best single strategy.The four-way ensemble reaches 51.70 and the five-way ensemble 52.98, versus 43.40 for best solo Query2Doc.
  • Enterprise result: +13.8 HIT@10: the five-way ensemble reaches 52.98 versus 39.22 for the S1 dense baseline.Budget-matched controls attribute only about 40% of the gain to retrieval depth, leaving complementarity as the main explanation.
  • Failure condition: −2.4pp: combining methods can hurt on AmbigNQ when their retrieved documents overlap heavily.The difference is statistically significant at p<0.001.
  • Cost-aware routing: +4.3 HIT@10: a confidence-gated router captures about half of the enterprise full-merge benefit while using rewriting on under 40% of queries.The router also improves downstream answer F1 by +1.92 and self-disables on AmbigNQ.

Limitations

The evaluation leaves important deployment and generalization questions unresolved, especially because the ensembles are post-hoc, the router is simulated, and several results rely on narrow settings.

  • Deployment scope: Production validation remains necessary because the ensembles are post-hoc analyses and the router is a simulated heuristic.The paper specifically calls for A/B testing and latency and cost measurements under load.
  • Evaluation scope: The answer-quality study uses one reader model and 200 queries from one benchmark, while SQuAD-style F1 only partially captures long-form answer quality.This limits how broadly the downstream quality result should be interpreted.
  • Generalization: Enterprise findings come from one synthetic benchmark, so generalization to other enterprise corpora is untested.The paper identifies this as an open boundary for the reported enterprise results.
  • Router limitations: The router is evaluated post-hoc with one fixed threshold; learned gating and live deployment studies remain future directions.The authors expect these extensions could further improve the cost–accuracy trade-off.

B Failure Modes and Rescue Rates

Failure analysis measures how often methods rescue baseline misses, showing that multi-variation strategies can recover substantial failures despite modest solo gains. These strategies also increase rescue rates when mixed.

  • Failure and rescue metrics: Even S4, despite modest solo gains, rescues a substantial fraction of S1 failures.This rescue behavior motivates the paper’s complementarity argument.
  • Failure and rescue metrics: Table 7 reports catastrophic-miss rates and rescue rates among queries missed by S1.Rescue rates are averaged over three seeds on a fixed seed-42 S1-miss set.
  • Failure and rescue metrics: S2/S3/S4 rescue-rate results reflect multi-variation mixing, while strategies are sorted by rescue rate for each dataset.The accompanying runtime table separately identifies the cost of these mixed strategies.

D Full Metrics

The reported metrics include additional retrieval results, experiment coverage, budget-matched ensemble comparisons, router sensitivity, and downstream answer quality. A separate retriever check indicates that the pipeline’s strength is not specific to BGE, although those runs are indicative rather than directly comparable.

  • Additional retrieval metrics: Table 9 reports additional retrieval metrics, with S2, S3, and S4 using multi-variation mixing.The supplied table text does not enumerate the individual metric values.
  • Experiment coverage: All 54 main experiment cells were completed across the reported coverage, including budget, retriever-robustness, and LLM-robustness conditions.The coverage table defines multi-variation mixing and distinguishes single-variation baselines.
  • Budget and router analyses: Budget-matched methods capture approximately 40% of the four-way ensemble gain on Enterprise HIT@10.The comparison attributes the remaining gain to complementarity rather than retrieval depth.
  • Budget and router analyses: Router sensitivity reports expansion rates and HIT@10 across thresholds, with always-merge behavior at τ≥0.75.Calibration transfers across datasets with a worst gap of 2.55pp and held-out-query gaps of at most +0.11pp.
  • Downstream answer quality: The downstream answer-quality study uses a 200-query Enterprise sample with GPT-4.1, where F1 is informative because EM is approximately zero.The router’s F1 gain concentrates on 75 router-triggered queries (+4.79, p<0.01), while non-triggered queries show no change.
  • Retriever robustness: With MiniLM, S1 reaches HIT@10 = 91.66 on HotpotQA and 96.00 on AmbigNQ, versus BGE’s 97.13 ± 0.11 and 93.67 ± 0.03.These are single-seed runs on an earlier February corpus build, so they are indicative rather than directly comparable.

K Experiment Configuration

The experiments use a fixed retrieval, reranking, diversification, chunking, mixing, filtering, generation, and seed configuration across the evaluated methods.

  • Retrieval pipeline: All runs use bge-base-en-v1.5 embeddings, bge-reranker-base, MMR λ=0.85, and sentence chunks of 512/50.These settings define the common retrieval and reranking pipeline.
  • Mixing and filtering: Score-based mixing keeps up to 20 merged documents, with retrieval depths K=10/7/5/8 for S1/S2/S3/S4.The configuration also applies cosine-based variant filtering in [0.4, 0.9].
  • Generation and seeds: LLM generation uses GPT-4.1 at temperature 0.7, with seeds 42, 123, and 2024.The same generation model and seed set are used across the reported runs.
Loading 2609.05637v1…