PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Rank in Generative Retrieval

Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li

arXiv:2306.15222v2cs.CLcs.AIcs.IR

TL;DR

Generative retrieval learns to generate passage identifiers, but that objective does not directly optimize the final passage ranking. LTRGR adds a learning-to-rank phase to align training with passage order, and outperforms other generative retrieval methods on three commonly used datasets.

  • Problem

    Generative retrieval optimizes identifier generation rather than passage ranking, and heuristic conversion of identifiers to ranked passages leaves a learning gap.

    Source: Introduction · p. 1

  • Method

    LTRGR adds a rank-loss training phase to autoregressive generative retrieval to optimize the model toward passage ranking.

    Source: Introduction · p. 2

  • Results

    LTRGR outperforms other generative retrieval methods on three commonly used datasets.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    LTRGR connects generative retrieval with classical learning to rank, creating opportunities for further research integrating the two paradigms.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Generative retrieval stands out as a promising new paradigm in text retrieval that aims to generate identifier strings of relevant passages as the retrieval target. This generative paradigm taps into powerful generative language models, distinct from traditional sparse or dense retrieval methods. However, only learning to generate is insufficient for generative retrieval. Generative retrieval learns to generate identifiers of relevant passages as an intermediate goal and then converts predicted identifiers into the final passage rank list. The disconnect between the learning objective of autoregressive models and the desired passage ranking target leads to a learning gap. To bridge this gap, we propose a learning-to-rank framework for generative retrieval, dubbed LTRGR. LTRGR enables generative retrieval to learn to rank passages directly, optimizing the autoregressive model toward the final passage ranking target via a rank loss. This framework only requires an additional learning-to-rank training phase to enhance current generative retrieval systems and does not add any burden to the inference stage. We conducted experiments on three public benchmarks, and the results demonstrate that LTRGR achieves state-of-the-art performance among generative retrieval methods. The code and checkpoints are released at https://github.com/liyongqi67/LTRGR.

Introduction

Generative retrieval trains models to generate passage identifiers, but its heuristic conversion to ranked passages leaves a gap between the generation objective and the ranking target. LTRGR integrates learning to rank into generative retrieval through an additional training phase.

Introduction

  • Generative retrieval generates passage identifiers as an intermediate target, then maps predicted identifiers to ranked passages; identifier strategies include numeric IDs and text spans.Numeric IDs require memorizing passage mappings, while text spans may correspond to multiple passages and require heuristic ranking.Source: Introduction · p. 1
  • Heuristic identifier-to-passage ranking uses sensitive hyperparameters outside the learning framework, and identifier-generation proficiency does not ensure an optimal passage order.This disconnect between the autoregressive learning objective and the desired ranking target creates a learning gap.Source: Introduction · p. 1
  • LTRGR first trains an autoregressive model to generate identifiers, then uses a rank loss over heuristic-mapped passage lists to optimize passage order.It adds a learning-to-rank training phase while retaining typical generative retrieval at inference.Source: Introduction · p. 2
  • LTRGR achieves state-of-the-art performance among generative retrieval methods on three widely used benchmarks, with no added inference burden.Source: Introduction · p. 2
  • LTRGR connects generative retrieval with classical learning to rank, aligning the learning objective with the desired passage-ranking target.Source: Introduction · p. 2

Related Work

Related work describes generative retrieval through passage identifiers, learning to rank through ranking-oriented training objectives, and dense retrieval as a prominent dual-encoder approach. The paper positions LTRGR as combining generative retrieval with learning to rank.

Generative Retrieval

  • Generative retrieval generates passage identifier strings as retrieval targets, while multiview identifiers represent passages from different perspectives.Source: Generative Retrieval · p. 2

Learning to Rank

  • Learning-to-rank methods include point-wise, pair-wise, and list-wise approaches; list-wise methods preserve ranking-list structure and can incorporate ranking measures into loss functions.Source: Learning to Rank · p. 2

Dense Retrieval

  • Dense retrieval is described as the de facto document-retrieval implementation, while generative retrieval may address missing fine-grained interaction but still lags state-of-the-art dense retrieval.Source: Dense Retrieval · p. 2

Method

LTRGR first trains an autoregressive model to generate multiview passage identifiers, then adds passage-ranking training to align learning with the final ranking target. The framework retains the generation objective and adds no inference-stage burden.

Learning to Generate

  • The autoregressive model learns to generate title, random-substring, or pseudo-query identifiers from a query and an identifier-view prefix.Training shuffles samples from the three views.Source: Learning to Generate · p. 3
  • At inference, predicted identifiers are converted into passage scores by summing the language-model scores of identifiers associated with each passage.The FM-index can locate passages containing predicted identifiers, avoiding scoring the entire corpus.Source: Learning to Generate · p. 3

Learning to Rank

  • Retrieved passages are ranked with margin-based losses comparing positive and negative examples, using both highest-scored and randomly sampled passages while retaining the generation loss.The ranking phase continues training the autoregressive model on passages retrieved after the generation phase.Source: Learning to Rank · pp. 3–4

Experiments

Experiments use open-domain QA and web-search benchmarks with full-corpus evaluation, comparing generative retrieval methods against established retrieval baselines.

Datasets

  • NQ and TriviaQA use natural-language questions and Wikipedia passages in the DPR setting, with evaluation over the full corpus.The study uses widely used benchmark metrics.Source: Datasets · p. 4
  • MSMARCO evaluates web-search queries against web-page passages in a Web search scenario.The experiments evaluate generative retrieval methods on this dataset.Source: Datasets · p. 4

Baselines

  • Generative baselines include DSI, DSI (scaling up), NCI, SEAL, and MINDER; BM25, DPR, and GAR provide additional comparisons.Baseline results were taken from the respective papers.Source: Baselines · p. 4

Implementation Details

Implementation continues training MINDER with BART-large as the backbone, using a three-epoch learning-to-rank phase.

Implementation Details

  • BART-large is the backbone, and the trained MINDER autoregressive model is continued with the learning-to-rank framework.This setup was chosen for comparison with previous work.Source: Implementation Details · p. 5
  • The ranking phase uses Adam with a 1e-5 learning rate, batch size 4, and three epochs.Training ran on one NVIDIA A100 GPU with 80 GB of memory.Source: Implementation Details · p. 5
  • For each training query, the top 200 retrieved passages supply positive and negative examples; training retains 40 predicted identifiers per passage.The margin m is 500 and weight λ is 1000.Source: Implementation Details · p. 5

Retrieval Results on QA

LTRGR improves on the prior best generative retrieval method on QA benchmarks, while comparisons with DPR vary by dataset and hit cutoff.

Retrieval Results on QA

  • 3.0 on NQ and 1.8 on TriviaQA: LTRGR improves hits@5 over MINDER, the previously best generative retrieval method.LTRGR is based on MINDER and adds a learning-to-rank phase.Source: Retrieval Results on QA · p. 5
  • On NQ, MINDER outperforms DPR across hits@5, hits@20, and hits@100, marking the first such generative-retrieval result under the full-corpus setting.The passage reports MINDER as the best method across those metrics.Source: Retrieval Results on QA · p. 5
  • On TriviaQA, LTRGR beats DPR at hits@100 but trails at hits@5 and hits@20; the authors suggest identifier-only retrieval and autoregressive error accumulation may explain the gap.The authors also note that MINDER performs significantly worse than DPR on TriviaQA.Source: Retrieval Results on QA · p. 5

Retrieval Results on Web Search

On MSMARCO, LTRGR delivered the strongest reported search performance, while several generative retrieval systems lagged BM25. The authors attribute this pattern as likely related to the lower quality and missing titles of web-sourced passages.

Retrieval Results on Web Search

  • The MSMARCO comparison labels model sizes to support fair comparisons because larger models typically perform better.Source: Retrieval Results on Web Search · p. 5
  • LTRGR outperformed all baselines, beating DSI (scaling up) by 5.7 points in MRR@10 and improving MINDER by 10.7 points in Recall@5 and 6.9 points in MRR@10.The authors report that LTRGR requires only an additional training step on MINDER.Source: Retrieval Results on Web Search · p. 5
  • SEAL, NCI, and DSI underperformed BM25, while MINDER and DSI (T5-large) only slightly outperformed it in the search scenario.The authors suggest lower-quality web passages and missing titles may explain generative retrieval's weaker performance than on QA datasets.Source: Retrieval Results on Web Search · p. 5

Ablation Study

The ablations indicate that rank losses are central to LTRGR's gains, while the generation loss remains necessary during rank training. Removing either margin-based loss also reduces performance.

Ablation Study

  • The ablation varies the generation and rank-loss terms, including removing both rank losses, either margin loss, or the generation loss.The experiments ask whether gains come from rank learning or continued training, whether both margin losses are needed, and what happens with rank loss alone.Source: Ablation Study · pp. 5–6
  • Table 4 compares SEAL with SEAL-LTR on NQ, where SEAL-LTR applies LTRGR to SEAL.Source: Table 4 · p. 6
  • Figure 2 plots NQ retrieval performance against margin values and balance weight λ.Source: Figure 2 · p. 6
  • Removing rank losses and using generation loss alone does not significantly affect performance, supporting the rank objective as the primary source of LTRGR's improvement.The authors conclude that the current combined loss is the best choice for the learning-to-rank phase.Source: Ablation Study · p. 6
  • Removing either margin-based loss lowers performance; the two losses use different sample-mining strategies and let the model leverage more passages.The authors say this promotes diversity among passages used in the loss.Source: Ablation Study · p. 6
  • Removing the generation loss makes LTRGR underperform MINDER; the authors observed local minima with smaller scores assigned to all passages.They conclude the generation loss is necessary during learning-to-rank training.Source: Ablation Study · p. 6

In-depth Analysis

The analysis tests LTRGR's transfer to SEAL, alternative rank losses, inference cost, and sensitivity to margin and loss weighting. It reports better SEAL retrieval metrics, weaker results with infoNCE, and distinct effects of the training settings.

Generalization of LTRGR

  • 3.6, 1.9, and 0.1 points: LTRGR improved SEAL’s hits@5, hits@20, and hits@100, respectively.The authors replaced MINDER with SEAL and used the same learning-to-rank training.Source: In-depth Analysis · p. 6

List-wise loss

  • LTRGR with infoNCE performed worse than its margin-based version; one-epoch training and unnormalized passage scores were offered as possible reasons.The comparison is reported in Table 5.Source: In-depth Analysis · p. 6, Table 5 · p. 6

Inference speed

  • 135 minutes for MINDER-based LTRGR and 115 minutes for SEAL-based LTRGR: inference took those times on NQ with beam size 15 on one V100 GPU.The added training step does not affect inference speed; runtime depends on the underlying model and beam size.Source: In-depth Analysis · p. 6

Margin analysis

  • Margin 100 performed worse than other tested variants, while raising it from 200 to 500 yielded only slight, nonsignificant gains.The authors note that larger margins can help distinguish positive from negative passages but can make the objective hard to reach.Source: In-depth Analysis · p. 6

λ analysis

  • As generation-loss weight increased across 200–500, hits@100 changed little while hits@5 and hits@20 gradually decreased.Setting the weight to 0 produced a significant performance gap, which the authors say confirms the importance of generation loss.Source: In-depth Analysis · p. 7

Effectiveness Analysis of Learning to Rank

On MSMARCO, the analysis attributes LTRGR’s gains mainly to positive passages moving higher in the ranking; a case study also reports more correct identifiers and higher scores for a shared identifier.

Quantitative analysis

  • Most gains came at top ranks: after LTR, positive passages shifted toward higher positions in MSMARCO retrieval lists.Figure 3(a) plots positive-passage counts by ranking position before and after learning-to-rank training.Source: Effectiveness Analysis of Learning to… · p. 7

Case Study

  • After learning-to-rank training, the case study showed more correct predicted identifiers and a higher score for the same identifier.The example identifier was “what is prime rate in Canada.”Source: Effectiveness Analysis of Learning to… · p. 7

Conclusion

LTRGR adds a passage rank-loss training step to generative retrieval without adding inference burden. Experiments show it outperforms other generative retrieval methods on three commonly used datasets.

Conclusion

  • LTRGR enables generative systems to learn to rank passages through an additional training step, with no added inference burden.The framework connects generative retrieval with classical learning-to-rank.Source: Conclusion · p. 7
  • LTRGR outperformed other generative retrieval methods on three commonly used datasets.Source: Conclusion · p. 7
  • The authors anticipate that deeper integration of generative retrieval and learning-to-rank will advance generative retrieval.Source: Conclusion · p. 7

Related papers

Loading 2306.15222v2…