PaperlayerView exact revision on arXiv

Source-linked AI summary

LURE-RAG: Lightweight Utility-driven Reranking for Efficient RAG

Manish Chandra, Debasis Ganguly, Iadh Ounis

arXiv:2601.19535v1cs.IR

TL;DR

RAG relevance retrieval can misalign with the utility of passages for downstream generation, while existing utility-driven methods may be inefficient and ignore document ordering. LURE-RAG adds an efficient LambdaMART reranker to any black-box retriever and trains it with listwise LLM-utility supervision. It reaches 97-98% of a strong dense baseline, while UR-RAG improves over the best existing baseline by up to 3%.

  • Problem

    Relevance-based retrieval may not reflect whether passages improve downstream generation, and existing utility-driven methods overlook ranking order or require dense, resource-intensive approaches.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    LURE-RAG adds a lightweight LambdaMART reranker to a black-box retriever and trains document ordering with listwise utility supervision from LLM outputs.

    Source: §1 · p. 2, §3.2 · p. 6, §3.4 · p. 7

  • Results

    LURE-RAG reaches 97-98% of the strongest dense neural reranker baseline, while UR-RAG improves accuracy and F1 by up to 3%.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    Utility-driven ranking losses provide an efficient and practical way to align retrieval ordering with downstream generation tasks.

    Source: §1 · p. 2, §6 · p. 13

    RePlug remains dependent on dense retrievers such as Contriever, limiting its direct applicability to BM25 and other non-parametric retrievers.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Most conventional Retrieval-Augmented Generation (RAG) pipelines rely on relevance-based retrieval, which often misaligns with utility -- that is, whether the retrieved passages actually improve the quality of the generated text specific to a downstream task such as question answering or query-based summarization. The limitations of existing utility-driven retrieval approaches for RAG are that, firstly, they are resource-intensive typically requiring query encoding, and that secondly, they do not involve listwise ranking loss during training. The latter limitation is particularly critical, as the relative order between documents directly affects generation in RAG. To address this gap, we propose Lightweight Utility-driven Reranking for Efficient RAG (LURE-RAG), a framework that augments any black-box retriever with an efficient LambdaMART-based reranker. Unlike prior methods, LURE-RAG trains the reranker with a listwise ranking loss guided by LLM utility, thereby directly optimizing the ordering of retrieved documents. Experiments on two standard datasets demonstrate that LURE-RAG achieves competitive performance, reaching 97-98% of the state-of-the-art dense neural baseline, while remaining efficient in both training and inference. Moreover, its dense variant, UR-RAG, significantly outperforms the best existing baseline by up to 3%.

1 Introduction

LURE-RAG addresses the mismatch between query relevance and downstream generation utility by adding a lightweight reranker trained with listwise utility supervision. It remains efficient while approaching dense neural reranker performance, and its dense variant UR-RAG significantly outperforms competing baselines.

1 Introduction

  • Semantic relevance does not necessarily indicate utility because retrieved documents can introduce incoherence, inconsistency, or omit facts needed for correct generation.Source: §1 · p. 2
  • Listwise ranking is needed because document order affects RAG generation, while existing utility-driven methods overlook comparative ranking and incur substantial training or inference costs.Source: §1 · p. 2
  • LURE-RAG adds a lightweight reranker between a black-box retriever and generator, using LLM utility to train document ordering.The framework treats both retriever and LLM as black boxes and uses listwise ranking supervision derived from how documents affect generation quality.Source: §1 · p. 2
  • 97-98% of the strongest dense neural reranker baseline is achieved by LURE-RAG while retaining modest training and inference overhead.Source: §1 · p. 2
  • UR-RAG, the dense neural counterpart, significantly outperforms competing baselines on multiple knowledge-intensive tasks.Source: §1 · p. 3

2 Related Work

Prior RAG methods often optimize relevance or utility with dense retrievers and expensive LLM-based scoring, while learning-to-rank methods provide ranking-aware objectives. LURE-RAG combines utility supervision with efficient LambdaMART reranking for black-box retrievers.

Relevance-based RAG

  • RAG combines retrieval of ranked documents with an LLM generator, while conventional approaches commonly rank passages using lexical or semantic relevance.Source: §2.1 · p. 3

Supervised Retriever (Utility-driven RAG)

  • RePlug uses LLM utility signals but depends on dense retrievers, limiting direct optimization of BM25 and other non-parametric retrievers in resource-constrained settings.Source: §2.1 · pp. 3–4
  • LLM-based zero-shot utility judging can mitigate position bias but requires multiple inference-time calls to shuffle and aggregate document scores.Source: §2.1 · p. 4

Learning to Rank

  • LambdaMART combines LambdaRank gradients with boosted regression trees and is attractive for lightweight reranking because of its efficiency, scalability, and interpretability.Source: §2.2 · p. 4

Lightweight Learning to Rank

  • LURE-RAG's workflow retrieves candidates, estimates document utility from LLM predictions, retains top-k documents, and trains LambdaMART on the resulting ranking data.Source: Figure 1 · p. 5

3 Proposed Methodology

LURE-RAG adds a lightweight LambdaMART reranker to black-box retrieval, using LLM-derived utility and listwise ranking to order candidates for generation.

3 Proposed Methodology

  • The reranker learns a scoring function fθ(q, d) from query-document features so its induced ranking maximizes expected downstream utility.Each candidate receives a predicted score used to form the new ordering.Source: §3.1 · pp. 5–6, §3.3 · p. 6
  • Document utility measures how including a passage affects the LLM’s downstream task score, such as F1 or Exact Match, and supplies reranker supervision.The utility function is estimated from LLM outputs relative to the gold answer.Source: §3.1 · p. 5, §3.2 · p. 6
  • LambdaMART trains regression-tree ensembles with ranking gradients that account for the NDCG change caused by swapping document positions.Its predicted scores are optimized so candidate order aligns with utility-based ground-truth rankings.Source: §3.4 · p. 7
  • LURE-RAG reranks the N documents returned by a first-stage retriever, selects the top k, and uses them to construct the LLM prompt.The retriever and LLM remain black boxes, while LambdaMART performs the intermediate ranking.Source: §3 · p. 5, §3.5 · p. 7

4 Experiment Setup

The experiments evaluate utility-driven reranking across datasets, LLMs, retrievers, and baselines, including whether ranking-aware training improves downstream performance and whether rerankers transfer across models. They use NQ-Open and TriviaQA, compare LURE-RAG and UR-RAG with relevance-based and RePlug baselines, and report Accuracy and F1.

Research Questions

  • The research questions test utility-driven reranking, listwise ranking loss, efficiency against dense rerankers, and transfer across downstream LLMs.The study examines performance across different LLMs and datasets, comparison with RePlug, and whether utilities learned from one LLM transfer to others.Source: §4.1 · pp. 7–8

Datasets

  • Experiments use NQ-Open and TriviaQA open-domain QA datasets, with Wikipedia-based retrieval settings described for both benchmarks.NQ-Open uses 100-word Wikipedia passages from a 20 December 2018 dump; TriviaQA experiments use a standardized English Wikipedia corpus.Source: §4.1 · p. 8

Baselines

  • Baselines include 0-shot generation, k-shot retrieval using first-stage passages, and a black-box adaptation of RePlug with a fixed retriever and trained reranker.The adapted RePlug setting retains its KL-based objective while avoiding retriever updates and periodic index refreshes.Source: §4.2 · pp. 8–9

Methods Investigated

  • LURE-RAG uses Phi-3-mini-4K-instruct utilities to train a LambdaMART reranker, while UR-RAG replaces LambdaMART with fine-tuned SBERT.UR-RAG is introduced to assess ranking loss using a dense reranker; LURE-RAG and UR-RAG use utility-driven supervision.Source: §4.2 · p. 9

Evaluation Metrics and Configurations

  • Evaluation reports Accuracy and F1, with results presented for BM25 and Contriever retrieval across dataset–LLM combinations.Accuracy accepts any predefined correct answer, while F1 rewards partial overlap; the experiments also vary LLM family and size.Source: §4.3 · pp. 9–10, Table 2 · p. 10, Table 3 · p. 11

5 Results

Across NQ and TQA, utility-driven reranking improves downstream performance over relevance-based retrieval, while LURE-RAG remains close to dense reranking with lower-cost LambdaMART models. UR-RAG further benefits from listwise ranking loss, and feature analysis identifies lexical statistics as important signals.

Comparison between relevance-based and utility-driven paradigms

  • Utility-driven approaches consistently outperform relevance-based retrieval across datasets and LLM choices, confirming that classical relevance alone is insufficient for downstream answer quality.The comparison spans NQ and TQA with llama-1B, phi-3.8B, and Qwen-14B.Source: §5 · p. 11

Effect of listwise ranking loss

  • UR-RAG significantly outperforms RePlug across all dataset–LLM combinations, including 0.2068 versus 0.1894 F1 on NQ with llama-1B and sparse retrieval.Paired t-tests at the 5% significance level find the improvements statistically significant across combinations; UR-RAG uses a ranking-aware objective while RePlug does not.Source: §5 · pp. 11–12

Comparison between dense and light-weight reranker

  • LURE-RAG reaches at least 97.8% of RePlug’s NQ F1 and 97.4% of its accuracy, with corresponding TQA ratios of 98.1% and 98.05%.These results support a lightweight LambdaMART reranker as a more efficient alternative with minor effectiveness trade-offs relative to the dense baseline.Source: §5 · p. 12
  • UR-RAG achieves the strongest accuracy and F1 scores across dataset–model combinations, while LURE-RAG remains above 90.3% of UR-RAG’s NQ F1 and 96.2% of its accuracy.For example, UR-RAG reaches 0.4186 accuracy and 0.4239 F1 on TQA with Qwen-14B and sparse retrieval.Source: §5 · p. 12

Cross-LLM generalization

  • A reranker trained on phi-3 utilities also performs well with downstream LLMs of different sizes, indicating transfer across generator models.The same trained reranker is reused across the evaluated LLMs.Source: §5 · p. 12

Interpretability

  • BM25 similarity is the most influential LambdaMART feature, while document-term counts and document-level IDF statistics also contribute notable importance.Feature importance is computed using gain, the summed improvement in the tree objective from splits using each feature; removing topic features causes only marginal degradation.Source: §5 · pp. 12–13, Figure 2 · p. 13

6 Conclusions

The paper concludes that LURE-RAG offers an efficient utility-driven reranking alternative by incorporating a ranking-aware objective, while UR-RAG achieves the strongest results. Future work will investigate richer semantic feature representations.

  • LURE-RAG improves RAG with a lightweight utility-driven reranking framework that incorporates a ranking-aware objective and offers an efficient alternative to existing approaches.Source: §6 · p. 13
  • UR-RAG achieves the strongest results, improving accuracy and F1 by up to 3%.Source: §6 · p. 13
  • Future work will explore richer feature representations, including semantic signals.Source: §6 · p. 13

Related papers

Loading 2601.19535v1…