PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning for Generative Retrieval over Dynamic Corpora

Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, Xueqi Cheng

arXiv:2308.14968v2cs.IR

TL;DR

Generative retrieval faces the challenge of adding documents to changing corpora while retaining access to old and new relevant documents. CLEVER combines incremental product quantization with memory-augmented learning, and experiments report effectiveness and efficiency for continual learning.

  • Problem

    Generative retrieval has largely been studied with fixed collections, although practical corpora gain documents over time and often have few labeled query-document pairs for new material.

    Source: §1 · p. 1

  • Method

    CLEVER combines partial, threshold-guided product-quantization updates for new docids with memory of similar old documents and generated pseudo-queries.

    Source: §1 · p. 2, §3.1.1 · p. 4

  • Results

    Experiments report that CLEVER is significantly better than prevailing continual-learning methods and demonstrate the method’s effectiveness and efficiency.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The experiments support CLEVER as an effective and efficient approach to continual learning when documents are added to a generative-retrieval corpus.

    Source: §7 · p. 9

    GR scalability remains challenging, especially for document addition, removal, and updates; proposed future work includes joint optimization and improved adaptive thresholds.

    Source: §7 · p. 9

Abstract

from arXiv · show

Generative retrieval (GR) directly predicts the identifiers of relevant documents (i.e., docids) based on a parametric model. It has achieved solid performance on many ad-hoc retrieval tasks. So far, these tasks have assumed a static document collection. In many practical scenarios, however, document collections are dynamic, where new documents are continuously added to the corpus. The ability to incrementally index new documents while preserving the ability to answer queries with both previously and newly indexed relevant documents is vital to applying GR models. In this paper, we address this practical continual learning problem for GR. We put forward a novel Continual-LEarner for generatiVE Retrieval (CLEVER) model and make two major contributions to continual learning for GR: (i) To encode new documents into docids with low computational cost, we present Incremental Product Quantization, which updates a partial quantization codebook according to two adaptive thresholds; and (ii) To memorize new documents for querying without forgetting previous knowledge, we propose a memory-augmented learning mechanism, to form meaningful connections between old and new documents. Empirical results demonstrate the effectiveness and efficiency of the proposed model.

1 INTRODUCTION

Dynamic corpora require generative retrieval systems to add documents over time while still retrieving both old and new relevant documents. CLEVER addresses this with incremental docid assignment and memory-augmented learning.

1 INTRODUCTION

  • Because GR stores document information in model parameters, adding documents otherwise requires costly retraining from scratch after each corpus update.Traditional IR can add new document representations to an external index without updating the retrieval model.Source: §1 · p. 2
  • CLEVER targets incremental indexing of new documents while retaining the ability to query both newly encountered and previously learned documents.The paper frames indexing cost and preserving retrieval ability as its two key challenges.Source: §1 · p. 2
  • Incremental product quantization updates selected centroids using two adaptive distance thresholds, assigning new documents docids without changing existing indices.The method supports unchanged, changed, or added centroids and maps document content to its docid.Source: §1 · p. 2
  • A dynamic memory bank preserves old exemplars similar to incoming documents, while a query generator supplies pseudo-queries during continual updates.These mechanisms are proposed to preserve previously indexed documents and retrieval ability.Source: §1 · p. 2
  • CLEVER is reported to mitigate catastrophic forgetting in incremental scenarios and outperform traditional IR and existing GR models in non-incremental scenarios.The introduction also reports significantly better performance than prevailing continual-learning methods.Source: Figure 1 · p. 2, §1 · p. 2

2 PROBLEM STATEMENT

The task begins with a labeled base collection, then adds document-only datasets in sequential sessions. The paper evaluates retrieval through both a shared test-query set and session-specific test sets.

2 PROBLEM STATEMENT

  • The continual-learning setup starts from a large base document set with labeled query-document pairs, followed by sequential sessions containing newly encountered documents without related queries.At each update, model parameters are trained using the new session and preceding datasets.Source: §2 · p. 2
  • With a single test-query set, the paper compares methods’ overall retrieval performance within the same session as relevant documents accumulate across sessions.It uses a common evaluation metric over queries and relevant documents from sessions up to the current session.Source: §2 · p. 2
  • With session-specific test-query sets, performance can be compared across sessions, using average performance, backward transfer, and forward transfer as additional measures.These measures capture end-of-training average performance, effects on preceding sessions, and learning on a new session, respectively.Source: §2 · p. 2

3 METHODOLOGY

CLEVER incrementally indexes documents using partially updated quantization codebooks, then combines indexing with memory-based rehearsal and pseudo-query training to preserve retrieval ability.

3 METHODOLOGY

  • CLEVER first indexes newly arrived documents, then protects retrieval ability during incremental indexing.Source: §3 · p. 2

3.1 Indexing new documents

  • New documents need low-cost docids and learned document–docid associations; CLEVER uses IPQ to represent docids in dynamic corpora.Source: §3.1 · p. 3, §3.1.1 · p. 3

3.1.1 Incremental product quantization.

  • IPQ bootstraps discriminative document representations, then uses two adaptive thresholds to update, preserve, or add centroids without changing old docid indices.Source: §3.1.1 · pp. 3–4

3.1.2 Indexing objective.

  • MLE trains the GR model to maximize each document's docid likelihood, forming the indexing objective.Source: §3.1.2 · p. 4

3.2 Preserving retrieval ability

  • CLEVER replays similar old documents from a dynamic memory bank and adds generated query–docid pairs to retain retrieval ability.Source: §3.2 · pp. 4–5

3.3 Overall training objective

  • Session training combines indexing and retrieval objectives, with EWC regularizing parameter changes from the previous model.Source: §3.3 · p. 5

4 EXPERIMENTAL SETTINGS

The experimental setup defines two continual-document benchmarks, compares traditional and generative retrieval baselines and model variants, and evaluates retrieval with MRR@10 and Hits@10. It also reports implementation choices for CLEVER and its baselines.

4.1 Benchmark construction

  • CDI-MS and CDI-NQ derive from MS MARCO 300k and Natural Questions; 60% of documents form D0, followed by four 10% increments, with dev-set evaluation and single or sequential query sets.The benchmarks use 300k MS MARCO documents and NQ data with 231k documents and 7.8k dev queries.Source: §4.1 · p. 5

4.2.1 Baselines.

  • Traditional baselines use BM25 re-indexing and DPR’s initial-session encoder, while DSI, DSI-QG, NCI, and Ultron are evaluated only in non-incremental scenarios.NCI is paired with DocT5Query, and Ultron uses product-quantized docids.Source: §4.2.1 · p. 5
  • BASE continually fine-tunes a PQ-docid retriever, whereas DSI++ continually fine-tunes DSI with atomic integer docids assigned to new documents.The authors re-implement DSI++ because its source code was not released.Source: §4.2.1 · p. 5

4.2.2 Model variants.

  • Quantization variants compare atomic docids, fixed PQ, per-session re-clustering, discriminative PQ, adaptive thresholds, and rebuilding codes for all documents.CLEVER_PQ uses BERTbase representations with a fixed codebook; CLEVER_PQ+Re re-clusters as documents arrive. The discriminative variants add ad and md thresholds separately or rebuild codes each session.Source: §4.2.2 · pp. 5–6
  • Memory variants remove EWC or memory-augmented losses, or replace CLEVER’s similar-document memory bank with randomly selected old documents.The random-memory variant retrains the GR model using the same number of old documents as CLEVER’s similar-document bank.Source: §4.2.2 · p. 6

4.3 Evaluation metrics

  • The reported continual-learning metrics are MRR@10 for CDI-MS and Hits@10 for CDI-NQ; model ordering is reported as consistent on other IR metrics.The paper also lists MRR@N, recall, hit ratio, and top-N retrieval accuracy as commonly used IR metrics.Source: §4.3 · p. 6

4.4 Implementation details

  • IPQ uses 24-code PQ vectors with 256 clusters and 768-dimensional representations, while the generative models use 12-layer, 768-dimensional encoder-decoder architectures.The document encoder starts from BERT; generative models start from T5base. Inference uses constrained beam search with 24 timesteps and 15 beams. The section also notes that code is available.Source: §4 · p. 5, §4.4 · p. 6

5 EXPERIMENTAL RESULTS

The experiments compare CLEVER and baselines across incremental and non-incremental retrieval, forgetting, forward transfer, and efficiency. CLEVER achieves the strongest reported results while largely mitigating forgetting and retaining low memory and training costs.

5.1 Baseline comparison

  • Re-clustering old and new documents performs worst because changing old centroids changes their docids, while updating old centroids or adding new ones improves assimilation.Source: §5.1 · p. 6
  • Pseudo-query/docid rehearsal and limiting model-update scope contribute to retrieval retention; random old-document selection may add noise, and the full CLEVER mechanism performs best.Source: §5.1 · pp. 6–7
  • CLEVER achieves the best single-query incremental performance and the best across-session AP, BWT, and FWT, with model ordering consistent across query settings.Source: §5.1 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
  • In non-incremental evaluation, CLEVER and other generative retrieval methods outperform traditional IR models, and CLEVER exceeds existing generative retrieval methods.The authors attribute CLEVER's advantage to its two-step iterative process for learning discriminative PQ codes as docids.Source: §5.1 · pp. 7–8, Table 3 · p. 7

5.2 Assessing catastrophic forgetting

  • CLEVER almost avoids catastrophic forgetting on indexing and retrieval, whereas applying IPQ or memory-augmented learning alone does not sufficiently preserve performance.Source: Figure 3 · p. 8, §5.2 · p. 8

5.3 Assessing forward transfer

  • CLEVER consistently and significantly outperforms independently fine-tuned INDIVIDUAL in the last four sessions, and also improves over CLEVERinit.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Effectiveness-efficiency trade-off

  • CLEVER achieves the best effectiveness with training times similar to BASE, supporting the authors’ conclusion that it is deployable in practical environments.Source: §5.4 · pp. 8–9
  • CLEVER has the strongest effectiveness while using only a small amount of additional memory over BASE, and that additional memory does not grow across sessions.Source: §5.4 · p. 8, Figure 4 · p. 9

6 RELATED WORK

This section reviews generative retrieval’s indexing and retrieval approaches, then turns to continual learning for dynamic corpora. Prior continual GR work leaves challenges in efficient docid assignment and finding useful relationships between new and old documents.

6 RELATED WORK

  • GR moves retrieval into a seq2seq model that maps queries to relevant docids, replacing the external index with model parameters.The model jointly supports indexing documents under docids and retrieving relevant docids from queries.Source: §6 · p. 9
  • Indexing research generates docids such as atomic IDs, titles [44], n-grams, URLs [54], and structured identifiers [54].Other work strengthens document-to-docid mappings with content at different semantic granularities and hyperlink information.Source: §6 · p. 9
  • Retrieval methods commonly map queries to docids autoregressively, while pseudo-queries [54] and pre-training tasks address limited labeled data.Source: §6 · p. 9
  • Mehta et al. [33] used arbitrary unique integer docids and replay [3], but individual embeddings were costly and random exemplars might miss new–old document relationships.Their evaluation considered only the sequential query-set setting.Source: §6 · p. 9
  • IPQ updates a subset of centroids without changing existing data indices; applying partial codebook updates to DSI’s clustering-based prefixes is future work.Source: §6 · p. 9
  • Continual learning addresses forgetting while acquiring knowledge from few labeled samples; this paper considers dynamic corpora with newly added documents.Source: §6 · p. 9
  • Continual-learning approaches include replay, regularization, and parameter isolation; this work uses replay and regularization.Source: §6 · p. 9

7 CONCLUSION

The paper presents continual learning for GR in corpora that gain documents, combining lower-cost docid generation with knowledge consolidation from similar past documents and pseudo-queries. Experiments demonstrate effectiveness and efficiency, while scalability for document addition, removal, and updates remains challenging.

7 CONCLUSION

  • CLEVER targets practical GR use as documents are continuously added, addressing docid-generation costs and consolidating knowledge with similar past documents and pseudo-queries.Source: §7 · p. 9
  • Extensive experiments demonstrate the method’s effectiveness and efficiency.Source: §7 · p. 9
  • GR scalability remains challenging for document addition, removal, and updates despite promising results.The paper identifies joint optimization of IPQ quantization and GR models using supervised labels, and advanced adaptive thresholds, as potential avenues for improving retrieval effectiveness.Source: §7 · p. 9

Related papers

Loading 2308.14968v2…