PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrieval

Guangyuan Ma, Xing Wu, Peng Wang, Zijia Lin, Songlin Hu

arXiv:2308.08285v1cs.IRcs.CL

TL;DR

The paper studies how to transfer LLM-generated query knowledge into dense passage retrievers without relying on LLM inference during retrieval. It develops contrastive and bottlenecked query-generation pre-training, finding stronger zero-shot and out-of-domain retrieval while curriculum learning reduces expansion needs.

  • Problem

    LLM-based document expansion offers enriched query context but is constrained by costly corpus-scale and online LLM inference, with limited methods addressing those costs.

  • Method

    The paper pre-trains dense retrievers with LLM-generated queries using contrastive learning, bottlenecked query generation, and a two-stage curriculum strategy.

  • Results

    Contrastive pre-training delivers stronger zero-shot and out-of-domain retrieval, while both pre-training paradigms achieve good retrieval after fine-tuning.

  • Takeaways & Limitations

    LLM-generated queries can support retrieval initialization without human-annotated data, while curriculum learning substantially reduces the number of required LLM-expanded queries.

  • Takeaways & Limitations

    The experiments do not cover additional LLM types or document-expansion prompts because the experiment budget is limited.

Abstract

from arXiv · show

In this paper, we systematically study the potential of pre-training with Large Language Model(LLM)-based document expansion for dense passage retrieval. Concretely, we leverage the capabilities of LLMs for document expansion, i.e. query generation, and effectively transfer expanded knowledge to retrievers using pre-training strategies tailored for passage retrieval. These strategies include contrastive learning and bottlenecked query generation. Furthermore, we incorporate a curriculum learning strategy to reduce the reliance on LLM inferences. Experimental results demonstrate that pre-training with LLM-based document expansion significantly boosts the retrieval performance on large-scale web-search tasks. Our work shows strong zero-shot and out-of-domain retrieval abilities, making it more widely applicable for retrieval when initializing with no human-labeled data.

Introduction

The paper addresses costly LLM-based document expansion by transferring generated-query knowledge into dense-retriever pre-training. It studies contrastive and bottlenecked query-generation strategies, with curriculum learning reducing LLM usage while improving zero-shot and fine-tuned retrieval.

  • Introduction: LLM-based document expansion can generate high-quality queries without human labels, but its online inference and corpus-processing costs limit practical use.The paper motivates pre-training retrievers so LLM inference is confined to pre-training rather than retrieval.
  • Introduction: The paper explores contrastive learning and bottlenecked query generation to incorporate LLM-generated queries into dense passage retrieval pre-training.Contrastive learning aligns query and passage representations, while bottlenecked generation transfers query context through an encoder representation.
  • Introduction: Contrastive pre-training improves in-domain zero-shot retrieval and out-of-domain adaptation, while bottlenecked query generation provides stronger initialization after fine-tuning.These findings are reported across MS-MARCO, TREC-DL, and BEIR evaluations.
  • Introduction: A two-stage curriculum reduces MS-MARCO passages requiring LLM inference from 8.8 million to 0.4 million with only minor performance degeneration.Randomly sampled passages initialize the retriever before a smaller LLM-expanded-query stage.

Methodology

The methodology uses dense passage retrieval representations enriched with LLM-generated queries. It combines query-generation prompts with contrastive or bottlenecked pre-training, while restricting LLM inference to retriever pre-training.

  • Preliminaries: Dense passage retrieval encodes queries and passages into representations whose similarity is measured by inner product or cosine distance.The paper frames stronger context alignment as the route to stronger query and passage representations.
  • Methodology: The framework uses LLM pseudo-queries with contrastive learning, bottlenecked query generation, and a curriculum that first initializes encoders from randomly sampled passages.The curriculum then introduces a smaller amount of LLM-expanded queries as fine-grained context.
  • LLM Query Generation: Alpaca-LLaMA receives zero-shot prompts and tk-Instruct receives few-shot prompts to generate queries for passages.The authors report that few-shot prompting stabilizes tk-Instruct generation, while Alpaca models work with zero-shot prompts.
  • LLM Query Generation: Generated queries enrich the pre-training corpus and are incorporated into end-to-end retriever pre-training rather than appended directly to passages or used during retrieval.This design confines LLM inference to the pre-training stage.

Bottlenecked Query Generation Pre-training

Bottlenecked query-generation pre-training compresses LLM-expanded context through an encoder and auxiliary decoder, while contrastive pre-training aligns passages with generated queries. The combined designs target strong initialization and retrieval performance with reduced expansion cost.

  • Bottlenecked Query Generation Pre-training: Bottlenecked query generation trains an encoder with MLM loss and an auxiliary decoder with CLM loss to reconstruct LLM-expanded queries from the encoder representation.The decoder receives the encoder’s bottlenecked representation and contextual text, while the encoder learns masked-token recovery.
  • Contrastive Pre-training: Table 1 evaluates zero-shot contrastive pre-training across MS-MARCO and TREC-DL 2019–2020, with statistically significant improvements over corresponding baselines reported for increments.The caption states that the best scores are bolded and significance is tested with two-tailed t-tests at p-value ≤0.01.
  • Bottlenecked Query Generation Pre-training: The bottlenecked encoder-decoder compresses query context into sentence embeddings, providing initialization for subsequent fine-tuning and inference.The decoder is connected through the encoder representation rather than exposing the full encoder sequence.

Contrastive Pre-training

Contrastive pre-training aligns passages with LLM-generated query contexts in a shared latent space while contrasting them against in-batch negatives. Its objective combines encoder, decoder, and contrastive losses.

  • Contrastive Pre-training: The coCondenser-style architecture forwards passages and sampled or generated contexts through the encoder, with an auxiliary decoder consuming encoder representations.The decoder is trained with a cross-entropy pre-task.
  • Contrastive Pre-training: LLM-generated queries are paired with passages in a contrastive objective that pulls their representations together and separates in-batch negatives.The final objective sums encoder MLM, decoder pre-task, and contrastive losses.
  • Contrastive Pre-training: The contrastive loss uses LLM-generated queries as passage contexts and in-batch context texts from other passages as negatives.This design directly aligns passage and query representations in the same latent space.

Curriculum Learning

The curriculum uses progressively richer contexts: randomly cropped spans first, followed by LLM-expanded queries. This substantially reduces LLM inference needs while maintaining similar retrieval performance.

  • Curriculum Learning: The two-stage curriculum pre-trains first on randomly cropped passage spans and then on fine-grained LLM-expanded queries.The first stage initializes the second-stage pre-training.
  • Curriculum Learning: The curriculum greatly reduces LLM inferences on MS-MARCO passages while maintaining similar retrieval performances.The approach treats cropped spans as coarse-grained contexts and generated queries as fine-grained contexts.

Zero-shot evaluation and Fine-tuning

The study evaluates zero-shot transfer and fine-tuning across MS-MARCO, TREC-DL, and BEIR, using contrastive retriever optimization and reported significance testing for fine-tuned results.

  • Zero-shot evaluation and Fine-tuning: The evaluation tests zero-shot contrastive-pretrained encoders without fine-tuning and assesses both pre-training schemes for retrieval initialization through fine-tuning.Datasets include MS-MARCO, TREC-DL, and BEIR for zero-shot evaluation.
  • Zero-shot evaluation and Fine-tuning: The fine-tuning objective uses a contrastive loss over queries, positive passages, and negative passages.The positive and negative passages define the retrieval training signal.
  • Contrastive Pre-training: Fine-tuned results are reported in Table 2, with improvements over corresponding baselines tested using two-tailed t-tests at p-value ≤0.01.The table marks best scores in bold and identifies reproduced results with †.

Experiments

The experiments evaluate LLM-expanded-query pre-training across zero-shot, fine-tuned, and out-of-domain retrieval settings, using MS-MARCO, TREC-DL, and BEIR benchmarks. Both contrastive and bottlenecked approaches are compared with span-based and established retrieval baselines.

  • Pre-training: Pre-training uses MS-MARCO’s 3.2M documents and LLM-generated queries from Alpaca 7B, Alpaca 13B, or tk-instruct 3B.The experiments also test downstream MS-MARCO Passage Ranking and TREC-DL tasks with standard retrieval metrics.
  • Baselines: The study compares against span-based self-contained baselines, BM25, SimCSE, coCondenser, Contriever, Condenser, SimLM, RetroMAE, and CoT-MAE.Multi-task and hybrid-retrieval methods are excluded as beyond fair-comparison scope.
  • Zero-shot Evaluation: LLM-expanded-query contrastive pre-training improves in-domain zero-shot retrieval over randomly sampled-span baselines on MS-MARCO and TREC-DL 2019 and 2020.The results indicate strong zero-shot retrieval ability in-domain.
  • Fine-tuned Retrieval: LLM-expanded queries significantly improve fine-tuned retrieval over both pre-training methods’ baselines and counterparts.The comparisons cover contrastive pre-training and bottlenecked query generation pre-training.
  • Out-of-domain Evaluation: On out-of-domain BEIR zero-shot evaluation, LLM-expanded contrastive pre-training improves over its contrastive baseline and outperforms Contriever despite Contriever using the larger CCNET corpus.BM25 remains a strong baseline among the compared methods without human-labeled fine-tuning.

Extended Analyses

Scaling LLMs improves zero-shot contrastive retrieval, while curriculum learning reduces the amount of LLM-expanded queries needed by both pre-training paradigms. After fine-tuning, LLM size differences largely disappear.

  • Effects of Scaling up LLMs: Scaling LLMs from 3B to 13B improves zero-shot contrastive retrieval when their generated queries are integrated into pre-training.Alpaca 13B is selected for better zero-shot results and on-par fine-tuned performance.
  • Effects of Scaling up LLMs: The scaling advantage does not persist after fine-tuning, where the authors hypothesize that all tested LLMs provide sufficiently good retrieval initialization.The explanation is presented as a hypothesis rather than a demonstrated mechanism.
  • Effects of Curriculum Learning: Curriculum learning benefits both pre-training schemes while reducing reliance on LLM-expanded queries.It uses sampled spans for 75% of training steps, followed by a smaller fine-grained query stage.
  • Effects of Curriculum Learning: Bottlenecked query generation surpasses its baseline after fine-tuning with only 0.4 million LLM-expanded queries.Zero-shot contrastive pre-training continues improving as the number of fine-grained queries increases.

Related Works

Prior dense-retrieval pre-training uses span relationships, cropped spans, granularity alignments, or auxiliary bottlenecked decoders. LLM-based document expansion addresses expensive inference and limited exploration by incorporating generated queries into retrieval pre-training.

  • Pre-training for Dense Retrieval: The paper’s core contribution is incorporating LLM-generated expanded queries into contrastive and bottlenecked dense-retrieval pre-training.This is positioned as improving context alignment and zero-shot and fine-tuned retrieval performance.
  • Pre-training for Dense Retrieval: Earlier dense-retrieval methods use constructed span relationships, randomly cropped spans, multiple-granularity alignments, or auxiliary bottlenecked decoders.The paper relates its approach to Gao and Callan (2022) and Wu, Ma, and Hu (2022).
  • LLM-based Query and Document Expansion: Traditional expansion uses query rewriting or fine-tuned T5 and BART models, while recent LLM query expansion enhances lexical query-passage matching.These approaches differ from the paper’s document expansion through query generation for pre-training.
  • LLM-based Query and Document Expansion: LLM-based document expansion remains underexplored because generating queries for huge document collections is expensive, especially for online inference.The paper proposes pre-training and curriculum learning to address these costs and can combine its method with traditional expansion at retrieval time.

Conclusion

The paper finds that LLM-based document expansion strengthens dense retrieval through contrastive and bottlenecked pre-training, including zero-shot and out-of-domain settings. Curriculum learning reduces required LLM-expanded queries, supporting unsupervised initialization without human labels.

  • Conclusion: LLM-based document expansion improves zero-shot and out-of-domain retrieval in contrastive pre-training and supports good fine-tuned retrieval with both proposed paradigms.The conclusion covers contrastive pre-training and bottlenecked query generation pre-training.
  • Conclusion: A two-stage curriculum strategy greatly reduces the need for LLM-expanded queries while causing only minor performance degeneration.The approach uses LLM-generated query knowledge during retriever pre-training rather than retrieval inference.
  • Conclusion: The paper concludes that LLM-based document expansion can initialize an unsupervised dense retrieval system when no human labels are available.The stated scope is retrieval initialization using LLM-generated enriched query context.

Limitation

The study does not evaluate additional LLM types, parameter sizes, or document-expansion prompts because the experiment budget was limited.

  • Additional LLM types, parameter sizes, and document-expansion prompts remain untested because the experiment budget was limited.The authors leave these investigations to future work.
Loading 2308.08285v1…