PaperlayerView exact revision on arXiv

Source-linked AI summary

Distillation Enhanced Generative Retrieval

Yongqi Li, Zhen Zhang, Wenjie Wang, Liqiang Nie, Wenjie Li, Tat-Seng Chua

arXiv:2402.10769v1cs.CLcs.AIcs.IR

TL;DR

Generative retrieval’s training can be hindered by incomplete judgments and binary relevance labels, while direct distillation transfer from dense retrieval is difficult. DGR trains generative retrieval with teacher-provided passage rankings and a distilled RankNet loss, achieving state-of-the-art results among generative retrieval methods on four datasets.

  • Problem

    Generative retrieval uses incomplete judgments and binary labels that treat all non-target passages as equally negative, while dense-retrieval distillation does not transfer directly across differing training paradigms.

    Source: §1 · pp. 1–2

  • Method

    DGR uses a teacher’s fine-grained passage rank order to train a generative retrieval model with a specially designed distilled RankNet loss.

    Source: §1 · p. 2

  • Results

    DGR achieves state-of-the-art performance among generative retrieval methods on four widely used datasets and is robust across teacher models and distillation losses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support distillation as a promising direction for improving generative retrieval systems.

    Source: §6 · p. 8

    DGR still falls short of state-of-the-art leaderboard performance; the authors attribute this to left-to-right generation potentially not capturing all passage content.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target. This paradigm leverages powerful generative language models, distinct from traditional sparse or dense retrieval methods. In this work, we identify a viable direction to further enhance generative retrieval via distillation and propose a feasible framework, named DGR. DGR utilizes sophisticated ranking models, such as the cross-encoder, in a teacher role to supply a passage rank list, which captures the varying relevance degrees of passages instead of binary hard labels; subsequently, DGR employs a specially designed distilled RankNet loss to optimize the generative retrieval model, considering the passage rank order provided by the teacher model as labels. This framework only requires an additional distillation step to enhance current generative retrieval systems and does not add any burden to the inference stage. We conduct experiments on four public datasets, and the results indicate that DGR achieves state-of-the-art performance among the generative retrieval methods. Additionally, DGR demonstrates exceptional robustness and generalizability with various teacher models and distillation losses.

1 Introduction

Generative retrieval’s training relies on incomplete, binary relevance judgments that can add noise and obscure differences in relevance. DGR uses teacher-provided passage rankings to train generative retrieval and achieves state-of-the-art performance among generative retrieval methods.

1 Introduction

  • Incomplete judgments and binary labels treat every non-target passage as equally negative, adding noise; dense-retrieval distillation also cannot transfer directly because generative retrieval trains differently.Source: §1 · pp. 1–2
  • DGR uses a stronger teacher to rerank initially retrieved passages, then trains the generative model with distilled RankNet on the fine-grained rank order; inference remains standard.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper introduces distillation as a direction for generative retrieval, with a loss designed to transfer knowledge from advanced ranking models.Source: §1 · p. 2
  • DGR achieves state-of-the-art performance among generative retrieval methods on four datasets and is robust across teacher models and distillation losses.Source: §1 · p. 2

2 Related Work

Related work describes generative retrieval’s use of passage identifiers and its remaining performance gap with advanced dense retrievers. It also situates DGR among text-retrieval distillation methods, noting that differences in training and architecture make direct transfer difficult.

2.1 Generative Retrieval

  • Generative retrieval represents passages with identifiers—including titles, numeric IDs, substrings, codebooks, and multiview identifiers—but still trails advanced dense retrievers.Source: §2.1 · p. 2

2.2 Knowledge Distillation in Text Retrieval

  • Prior text-retrieval distillation transfers cross-encoder knowledge into dense retrievers, but architectural and training differences make direct transfer to generative retrieval challenging; DGR proposes a feasible framework.Source: §2.2 · pp. 2–3

2.3 Dense Retrieval

  • Dense retrieval uses dual encoders and MIPS for efficient large-corpus retrieval, while generative encoder-decoder retrieval may address missing fine-grained interactions but still trails state-of-the-art dense methods.Source: §2.3 · p. 3

3 Method

Section 3 outlines MINDER’s generative retrieval setup and DGR’s use of teacher-provided passage rankings to train a generative retrieval model. DGR uses a rank-sensitive loss while retaining the original inference stage.

3.1 Preliminary: Generative Retrieval

  • MINDER represents each passage with a title, a randomly selected substring, and a content-based pseudo-query that complement one another across search scenarios.It trains an autoregressive language model to map query text to a relevant passage identifier, randomly shuffling the three identifier types during training.Source: §3.1 · p. 3
  • At inference, beam search generates identifiers, and a heuristic ranks passages by summing language-model scores for generated identifiers found in each passage.The passage score aggregates the scores of its selected, covered identifiers.Source: §3.1 · p. 3

3.2 Distillation enhanced Generative Retrieval

  • DGR uses a teacher model to rerank student-retrieved passages, treating the resulting order as knowledge that reflects varying relevance rather than binary labels.Cross-encoders are considered as teachers because they rank passages more powerfully than the student model.Source: §3.2 · pp. 3–4
  • DGR’s distilled RankNet loss compares student passage scores using margins that increase with the passages’ separation in the teacher ranking.The loss forms passage pairs and uses the teacher’s order to guide listwise optimization.Source: §3.2 · p. 4
  • Training interpolates generation and distillation losses, and the framework adds no burden to the original inference stage.After training, the autoregressive model retrieves passages as in the preliminary system.Source: §3.2 · p. 4

4 Experiments

Across QA and Web search benchmarks, DGR outperforms other generative retrieval methods. Its QA results surpass DPR on NQ and slightly on TriviaQA, while a gap to leading dense retrievers remains.

4.1 Setup

  • Experiments cover NQ and TriviaQA for QA and MSMARCO and TREC DL 2019/2020 for Web search, using dataset-specific retrieval metrics.NQ and TriviaQA use hits@5, @20, and @100; MSMARCO uses Recall and MRR; TREC DL uses nDCG.Source: §4.1 · p. 4
  • Comparisons include generative retrieval methods DSI, NCI, SEAL, MINDER, and LTRGR, alongside BM25, DPR, and GAR.Baseline results were taken from the respective papers.Source: §4.1 · p. 5

4.2 Retrieval Results on QA

  • DGR improves hits@5 over LTRGR by 2.6 on NQ and 2.4 on TriviaQA, and raises NQ hits@5 from 65.8 to 71.4 over MINDER.The Table 1 caption identifies the comparison as retrieval performance on NQ and TriviaQA.Source: Table 1 · p. 5, §4.2 · p. 6
  • DGR significantly outperforms DPR on NQ and slightly surpasses DPR on TriviaQA, although generative retrieval still trails state-of-the-art dense retrievers in the benchmarks.The Table 1 caption identifies the QA results as retrieval performance on NQ and TriviaQA.Source: Table 1 · p. 5, §4.2 · p. 6

4.3 Retrieval Results on Web Search

  • On Web search, DGR outperforms LTRGR, but most generative methods fall below BM25 and the leading MSMARCO and TREC DL methods.The Table 2 caption identifies results for MSMARCO and TREC and defines its reported metrics as Recall and MRR.Source: Table 2 · p. 5, §4.3 · p. 6

5 Analysis

Section 5 examines how distillation losses, teacher models, sampling strategies, and tuning choices affect DGR. The analyses report robustness across teachers and losses, identify an effective sampling mix, and describe practical limits on teacher-list length and inference cost.

5.1 Analysis on Distillation Loss

  • DGR improves the base model with different distillation losses, while distilled RankNet significantly enhances performance by using incremental margins from the teacher’s ranking list.The tested alternatives include KL divergence and pairwise and listwise losses; the experiments include ListMLE, ListNet, approxNDCG, RankNet, and LambdaLoss.Source: §5.1 · pp. 6–7

5.2 Analysis on Teacher Models

  • DGR improves retrieval with both cross-encoder and dual-encoder teachers, with similar gains across teacher models.The authors suggest the teacher list length of 6, limited by GPU resources, may not fully capture differences in ranking ability.Source: §5.2 · p. 7, Figure 2 · p. 7

5.3 Analysis on Sampling Strategies

  • Top&random achieves better overall hits@5, hits@20, and hits@100 than the other sampling strategies.It selects one top-ranked passage and samples the remaining M−1 passages randomly; considering only the top M passages is too extreme when M is 6.Source: §5.3 · p. 7, Table 5 · p. 7

5.4 In-depth Analysis

  • For the incremental margin, 100 performs better than 50, while performance gradually declines as the margin rises from 100 to 500.The authors note that larger margins may be unsuitable because many passages can be relevant to a query.Source: §5.4 · p. 8
  • Retrieval performance gradually increases as the number of passages M in the teacher list increases, but GPU memory limited the tested maximum to 8.A longer teacher list requires more GPU memory and represents more teacher-provided knowledge.Source: §5.4 · p. 8
  • The authors test DGR with SEAL as the base generative retrieval model and compare it with an additional training framework.The supplied passage describes the experiment and points to Table 6 but does not report its result values.Source: §5.4 · p. 8, Table 6 · p. 8
  • DGR, LTRGR, and MINDER each take approximately 135 minutes for inference on the NQ test set using one V100 GPU with 32GB memory; SEAL takes 115 minutes.DGR’s distillation loss does not impact the inference process of the underlying generative retrieval model.Source: §5.4 · p. 8

6 Conclusion and Future Work

The paper concludes that distillation can improve generative retrieval and presents DGR as adaptable across teacher models and losses. Future work includes additional teachers, sampling strategies, and progressive training.

6 Conclusion and Future Work

  • DGR distills knowledge from advanced rank models to improve generative retrieval, and the paper reports adaptability across teacher models and distillation losses.Experiments on four benchmarks are reported as evidence of DGR’s effectiveness and robustness; the authors describe distilled RankNet as more effective than existing distillation losses.Source: §6 · p. 8
  • Future work could explore advanced generative retrieval models as teachers, more sampling strategies, and progressive training methods.The authors say the paper has only begun validating distillation’s feasibility for generative retrieval.Source: §6 · p. 8

Limitations

The authors identify limits in teacher-list length and in DGR’s standing relative to broader retrieval leaderboards.

Limitations

  • GPU memory limited the assessed teacher-list length to 8, so possible gains from longer lists remain unverified.The authors note that longer lists typically provide more teacher knowledge.Source: Limitations · p. 9
  • Although DGR performs best among current generative retrieval methods, it falls short of current state-of-the-art leaderboard performance.The authors attribute this gap to autoregressive left-to-right generation, which may not fully capture a passage’s entire content.Source: Limitations · p. 9

Ethics Statement

The authors describe the datasets and artifacts used in the study and state that the work meets ACL ethics standards.

Ethics Statement

  • The experiments use publicly released datasets labeled through human interaction in English.The authors state that user privacy is protected and the datasets contain no personal information.Source: Ethics Statement · p. 9
  • The scientific artifacts used are available for research under permissive licenses.The authors state that their use in the paper is consistent with the artifacts’ intended use.Source: Ethics Statement · p. 9
  • The authors conclude that the research meets ACL ethics standards.Source: Ethics Statement · p. 9
Loading 2402.10769v1…