PaperlayerView exact revision on arXiv

Source-linked AI summary

DSI++: Updating Transformer Memory with New Documents

Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Jinfeng Rao, Marc Najork, Emma Strubell, Donald Metzler

arXiv:2212.09744v3cs.CLcs.AIcs.IRcs.LG

TL;DR

Updating a DSI for a changing corpus is computationally expensive because corpus updates require retraining from scratch. DSI++ studies incremental indexing as a continual-learning challenge and combines flatter-minima training with generative-memory rehearsal. On NQ and MS MARCO, generative-memory rehearsal improves retrieval metrics over competitive baselines, although the method exhibits significant forgetting on full MS MARCO.

  • Problem

    Updating a dynamic DSI corpus requires retraining the model from scratch, incurring prohibitively high computational costs.

    Source: §1 · p. 1

  • Method

    DSI++ studies incremental indexing as continual learning and uses SAM to promote flatter minima alongside generative memory that supplies pseudo-queries during indexing.

    Source: §1 · pp. 1–2

  • Results

    Generative-memory rehearsal improves average Hits@1 by +25.0% on MS MARCO and Hits@10 by +21.1% on NQ over competitive baselines, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The proposed solutions significantly reduce catastrophic forgetting in a setting where continuous document addition is needed for practical DSI use.

    Source: §6 · p. 9

    On the full MS MARCO dataset, the method exhibits significant forgetting.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Differentiable Search Indices (DSIs) encode a corpus of documents in model parameters and use the same model to answer user queries directly. Despite the strong performance of DSI models, deploying them in situations where the corpus changes over time is computationally expensive because reindexing the corpus requires re-training the model. In this work, we introduce DSI++, a continual learning challenge for DSI to incrementally index new documents while being able to answer queries related to both previously and newly indexed documents. Across different model scales and document identifier representations, we show that continual indexing of new documents leads to considerable forgetting of previously indexed documents. We also hypothesize and verify that the model experiences forgetting events during training, leading to unstable learning. To mitigate these issues, we investigate two approaches. The first focuses on modifying the training dynamics. Flatter minima implicitly alleviate forgetting, so we optimize for flatter loss basins and show that the model stably memorizes more documents ($+12\%$). Next, we introduce a generative memory to sample pseudo-queries for documents and supplement them during continual indexing to prevent forgetting for the retrieval task. Extensive experiments on novel continual indexing benchmarks based on Natural Questions (NQ) and MS MARCO demonstrate that our proposed solution mitigates forgetting significantly. Concretely, it improves the average Hits@10 by $+21.1\%$ over competitive baselines for NQ and requires $6$ times fewer model updates compared to re-training the DSI model for incrementally indexing five corpora in a sequence.

1 Introduction

DSI++ frames incremental indexing as a continual-learning challenge: add documents while retaining the ability to answer queries about old and new documents. It examines forgetting across models and docid representations and proposes training-dynamics and generative-memory approaches to mitigate it.

1 Introduction

  • Because DSI encodes corpus information in Transformer parameters, updating a dynamic corpus otherwise requires retraining the model from scratch at high computational cost.DSI maps document content to docids for memorization and user queries to relevant docids for retrieval.Source: §1 · p. 1
  • DSI++ frames incremental indexing as continual learning: add documents while preserving query access to old and new documents, with benchmarks based on NQ and MS MARCO.The benchmarks probe forgetting across T5-Base, T5-Large, and T5-XL, and atomic, naively structured, and semantically structured docids.Source: §1 · pp. 1–2
  • Continual indexing severely forgets previously memorized documents, while approximately 88% of documents experience a forgetting event after memorization.A forgetting event occurs when a document prediction changes from the correct docid to an incorrect one during learning.Source: §1 · p. 2, Figure 1 · p. 2
  • Generative memory samples pseudo-queries for indexed documents to alleviate retrieval forgetting and for incoming documents to support continual semi-supervised retrieval learning.This rehearsal approach complements the training-dynamics intervention by targeting the retrieval task during incremental indexing.Source: §1 · p. 2
  • SAM promotes flatter loss basins to mitigate forgetting during memorization and yields +12% more stably memorized documents.The paper hypothesizes and verifies that forgetting events occur throughout DSI memorization.Source: §1 · p. 2
  • A generative memory-based experience-rehearsal approach alleviates explicit forgetting during continual indexing and improves average Hits@1 by +25.0% and Hits@10 by +21.1% over competitive baselines on MS MARCO and NQ, respectively.Source: §1 · p. 2

2 DSI++: Continual learning challenge for DSI

DSI++ frames incremental indexing as a continual-learning problem: the model must index new document batches without losing the ability to answer queries about old and new documents. The case study finds severe forgetting, alongside differences by retrieval performance, docid representation, and model scale.

2.1 Problem setup

  • The system receives document batches D1 through Dt without queries for incoming documents, and must answer queries about both previously and newly indexed documents.The setup uses the same indexer for document-to-docid memorization and query-to-docid retrieval.Source: §2.1 · pp. 2–3

2.2 Benchmarks for DSI++

  • The NQ and MS MARCO benchmarks each use an initial 50K-document corpus followed by five sequential batches of 10K documents.NQ uses Wikipedia articles and questions; MS MARCO uses passages and passage-query pairs.Source: §2.2 · p. 3

2.3 Evaluation Metrics

  • The evaluation tracks indexing accuracy and Hits@1/10, plus average performance An, forgetting Fn, and forward-transfer learning performance LAn across corpora.Fn measures effects on previously indexed corpora; LAn averages performance on newly indexed corpora.Source: §2.3 · p. 3

2.4 Case study: Forgetting and Forward Transfer

  • After indexing D5, indexing-accuracy forgetting F5 reaches 75, and the initially indexed D0 loses about 25 indexing-accuracy points and 20 Hits@1 points.Forgetting worsens as corpora are added, and is severe across model scales.Source: §2.4 · p. 4, Figure 2 · p. 4
  • Retrieval learning performance LAn declines across sequential indexing, even as indexing-accuracy LAn increases, indicating different trends for memorization and retrieval.The authors attribute the retrieval decline to model drift under continual indexing updates.Source: §2.4 · p. 4, Figure 2 · p. 4
  • Naively structured docids score about 30 average Hits@1 versus 39 for atomic docids; semantic structure reduces forgetting relative to naive structure but still underperforms atomic docids.The study reports that naive structured docids are more prone to interference from new docids.Source: §2.4 · p. 4, Figure 2 · p. 4
  • T5-XL exceeds T5-Large and T5-Base in average An and learning LAn, but forgetting Fn remains severe at every scale without a clear best performer.The authors use T5-Base for subsequent experiments.Source: §2.4 · p. 4, Figure 2 · p. 4

3 Implicit Forgetting: SAM

The section identifies unstable memorization as implicit forgetting during training and tests whether flatter loss basins reduce it. SAM improves stable memorization over Adafactor, while the authors note that forgetting remains open for further improvement.

Forgetting events

  • About 88% of documents experience at least one forgetting event, defined as a document changing from correctly to incorrectly mapped during memorization.Fluctuating indexing accuracy throughout training motivates the authors' implicit-forgetting hypothesis.Source: §3 · p. 5

Sharpness-Aware Minimization

  • SAM optimizes for parameters in neighborhoods with uniformly low loss, using a perturbed-weight gradient update to seek flatter loss basins.The authors hypothesize that this training-dynamics change should alleviate implicit forgetting.Source: §3 · p. 5

SAM alleviates implicit forgetting

  • SAM stably memorizes 12% more documents with zero forgetting events than Adafactor, and scores Hits@1 of 35.9±2.2 versus 32.5±6.4 on D0.The comparison uses a pretrained T5-Base model and 50K unique documents.Source: §3 · pp. 5–6, Figure 3 · p. 5

Discussion

  • The results extend prior flatness-and-forgetting findings to document memorization without clear distributional shift, though the authors say improvement remains possible.The section compares SAM with Adafactor for memorizing D0.Source: §3 · pp. 5–6

4 Explicit Forgetting: Generative Memory

Continual indexing can impair retrieval, and incoming documents lack ground-truth queries for replay. Generative memory samples pseudo-queries for old and new documents to support retrieval-task replay and semi-supervised learning.

Explicit forgetting and retrieval task

  • Continual indexing causes retrieval forgetting across model scales and docid representations, while retrieval learning performance declines under indexing updates.The authors report that indexing old and new documents alone still leaves forgetting.Source: §4 · p. 6

Episodic memory

  • Episodic replay cannot use ground-truth queries for incoming documents, and retaining only initial-corpus queries suffers from forward transfer to newly indexed documents.The section motivates a parametric alternative to retaining ground-truth queries.Source: §4 · p. 6

Generative memory

  • Generative memory trains a query generator to sample pseudo-queries for previously seen documents and incoming batches, supplementing them during indexing for retrieval replay and continual semi-supervised retrieval learning.The method uses generated queries for sparse experience replay.Source: §4 · p. 6

5 Experimentation

Experiments compare continual-indexing strategies on NQ and MS MARCO. Generative memory reduces retrieval forgetting and supports transfer to new documents, while incremental indexing uses fewer updates than retraining from scratch.

5.1 Methods

  • The methods compare retraining from scratch, continual indexing of new or all seen documents, and replay using ground-truth queries or generated pseudo-queries.The proposed generative-memory method samples pseudo-queries and uses them for sparse retrieval-task replay.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5 Experimentation

  • Experiments evaluate incremental indexing on NQ and MS MARCO, including performance after updates across a sequence of five corpora.Models use pre-trained T5-Base initialization with randomly initialized atomic docid-token parameters.Source: §5 · p. 6, §5.2 · p. 7

5.2 Results

  • Generative memory reduces forgetting and improves forward transfer on NQ; replaying pseudo-queries for old and new documents supports both outcomes.Replay of queries for only D0 or D1 can hurt forward transfer or increase forgetting; across five updates, generative memory outperforms cl(U5) on average performance and forgetting.Source: §5.2 · pp. 7–8
  • +21.1% average Hits@10 over considered baselines when generative memory augments continual indexing on NQ.The reported comparison is for T5-Base with atomic docids; the method also reduces forgetting.Source: Figure 4 · p. 8
  • On MS MARCO, after training on D0 passages, continual finetuning on D1 with cl(D1) drops D0 Hits@1 from 16.3 to 0.1, while generative memory retains 7.3.With pseudo-query augmentation for D1 passages, generative memory yields D1 Hits@1 of 31.6 versus 18.2 for cl(D1).Source: §5.2 · p. 9
  • Slightly above 300K additional updates index NQ corpora D1–D5, approximately six times fewer than retraining, with superior overall performance.Retraining would require around 1.75M steps after the initial D0 training.Source: §5.2 · p. 9

6 Conclusion

DSI++ addresses the need to add documents to DSI corpora without repeatedly rebuilding the index from scratch.

6 Conclusion

  • DSI++ applies sharpness-aware minimization and generative memory to reduce catastrophic forgetting when documents are added continuously.The authors describe the work as a foundation for further research on DSI and continual semi-supervised learning.Source: §6 · p. 9

Limitations

The study's scope and performance have boundaries that warrant further investigation.

Limitations

  • The study examines additions of distinct documents; behavior is unpredictable when new documents refute or modify indexed ones, and significant forgetting remains on larger datasets.The authors call for further analysis of contradictory documents and improved performance at larger scales.Source: Limitations · p. 9

Ethics Statement

The paper frames incremental learning as a way to reduce retraining costs while noting that lower costs may increase consumption.

Ethics Statement

  • Continual learning requires many fewer training steps than retraining from scratch, but reduced cost can increase overall consumption through Jevons’ paradox.The authors discuss this trade-off in the context of training cost and environmental impact.Source: Ethics Statement · p. 10
  • The authors aim to make adding document sets cheaper and better for the environment by reducing the need to retrain DSI models from scratch.They report analyzing index-construction time in their experiments.Source: Ethics Statement · p. 10
  • The paper links training large models to environmental impact and cites Strubell et al. (2019) in that discussion.It also cites Jevons’ paradox to qualify the potential impact of reduced training cost.Source: Ethics Statement · p. 10

A.1 Implementation Details

The implementation initializes models from pre-trained T5 and uses SAM settings informed by prior work. Training schedules and checkpoint selection differ between initial indexing, continual indexing, and generative-memory training.

A.1 Implementation Details

  • T5-Base initializes the models, while additional atomic document-identifier token parameters are randomly initialized.The SAM setup uses ρ = 0.15 and an inner-maximization batch size of 32.Source: §A.1 · p. 12
  • Initial indexing runs for up to 1M steps with 100K warmup steps; continual indexing runs for up to 100K steps with 100 warmup steps.Both use learning rate 0.001, batch size 128, and input sequence length 32; checkpoints are evaluated every 5K steps.Source: §A.1 · p. 12
  • Initial-model selection averages indexing accuracy, Hits@1, and Hits@10, whereas continual-learning selection uses indexing accuracy alone.The continual-learning experiments have access only to indexing accuracy for the involved corpora.Source: §A.1 · pp. 12–13
  • Figure 5 contrasts Adafactor's fluctuating indexing accuracy with SAM's relatively stable memorization.The figure investigates SAM for T5-Base during memorization.Source: Figure 5 · p. 13
  • The parametric query generator trains on retrieval data R0 for up to 100K steps and uses beam decoding to generate pseudo-queries.Document contents have maximum length 1024 and generated queries target length 32; model selection uses BLUE.Source: §A.1 · p. 13

A.2 Related Work

Prior work frames continual learning as retaining past knowledge while learning new tasks. For DSI++, the cited work distinguishes training-time forgetting mitigation from test-time approaches and motivates pseudo-query generative memory for incremental indexing.

A.2 Related Work

  • Optimization-based continual-learning methods modify training dynamics; prior work links flatter minima and SAM to reduced forgetting, and DSI++ applies SAM to stabilize memorization.Figure 5 reports serious indexing-accuracy fluctuations with Adafactor and relatively stable memorization with SAM.Source: §A.2 · p. 13
  • On MS MARCO, Table 3 reports that continual indexing on updated corpora reduces forgetting versus indexing only new documents, while pseudo-query replay for all documents alleviates forgetting and improves forward transfer.The table also considers incremental indexing in the full MS MARCO setting with 8.9M passages.Source: Table 3 · p. 14
  • Across T5-Base, T5-Large, and T5-XL, forgetting remains severe; semantic docid structure alleviates forgetting, while naive string docids underperform atomic docids on Hits@10.The figure also reports improved average An and learning LAn performance with increasing model scale.Source: Figure 6 · p. 14
  • Unlike episodic memory, DSI++ learns a parametric generator to produce pseudo-queries for previously seen documents and supplement them during incremental indexing.The approach also uses unlabeled incoming documents for continual semi-supervised learning because queries for new documents are unavailable.Source: §A.2 · p. 14
  • For NQ and MS MARCO, generative memory reduces forgetting and improves average retrieval performance over continual indexing; the reported gains are +17.3% Hits@1 and +23.0% Hits@10, respectively.Continual indexing old and new documents alone still leaves average Hits@1 down 19 points on NQ and average Hits@10 down 25.0 points on MS MARCO after sequential updates.Source: Figure 7 · p. 15, Figure 8 · p. 15
  • The proposed pseudo-query approach is reported to outperform episodic memory, but generated queries may hallucinate unsupported content, motivating future filtering.The approach generates pseudo-queries rather than example pairs.Source: §A.2 · pp. 15–16
  • Test-time adaptation is computationally expensive when it updates the DSI indexer for every query, while retrieval-augmented generation does not assess forgetting in model parameters.DSI++ instead studies continual learning during training with corpus information encoded in model parameters.Source: §A.2 · p. 16
  • Parameter-isolation methods typically need task identity at inference, which conflicts with DSI's query-to-document-identifier prediction setup.The authors leave adapting those methods to DSI++ for future work.Source: §A.2 · p. 16

Related papers

Loading 2212.09744v3…