PaperlayerView exact revision on arXiv

Source-linked AI summary

CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks

Jiafeng Guo, Changjiang Zhou, Ruqing Zhang, Jiangui Chen, Maarten de Rijke, Yixing Fan, Xueqi Cheng

arXiv:2402.16767v1cs.IRcs.CL

TL;DR

KILT retrieval models have largely been studied with static corpora, despite continuing document arrivals. This paper defines continual document learning for KILTs, builds KILT++, and proposes CorpusBrain++; experiments report effective, efficient continual retrieval with reduced forgetting and positive forward transfer.

  • Problem

    Most KILT retrieval models focus on stationary knowledge sources, while real-world corpora continue to gain documents and CorpusBrain’s implicit index makes updating a challenge.

    Source: §1 · p. 2

  • Method

    The paper defines continual document learning for KILTs, constructs KILT++ from session-split KILT data, and proposes CorpusBrain++ for evolving corpora.

    Source: §1 · p. 3

  • Results

    On KILT++, CorpusBrain++ handles continual document learning effectively and efficiently, mitigates forgetting of previously encountered documents, and shows positive forward knowledge transfer.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The reported framework combines a backbone-adapter architecture, task-specific pre-training, and semantic-similarity document rehearsal for continual KILT retrieval.

    Source: §8 · p. 29

    The authors suggest CorpusBrain++ may be unstable with long, complex queries, including ELI5 questions, and identify query length and complexity in pre-training objectives as future work.

    Source: §6.1.1 · p. 20

Abstract

from arXiv · show

Knowledge-intensive language tasks (KILTs) typically require retrieving relevant documents from trustworthy corpora, e.g., Wikipedia, to produce specific answers. Very recently, a pre-trained generative retrieval model for KILTs, named CorpusBrain, was proposed and reached new state-of-the-art retrieval performance. However, most existing research on KILTs, including CorpusBrain, has predominantly focused on a static document collection, overlooking the dynamic nature of real-world scenarios, where new documents are continuously being incorporated into the source corpus. To address this gap, it is crucial to explore the capability of retrieval models to effectively handle the dynamic retrieval scenario inherent in KILTs. In this work, we first introduce the continual document learning (CDL) task for KILTs and build a novel benchmark dataset named KILT++ based on the original KILT dataset for evaluation. Then, we conduct a comprehensive study over the use of pre-trained CorpusBrain on KILT++. Unlike the promising results in the stationary scenario, CorpusBrain is prone to catastrophic forgetting in the dynamic scenario, hence hampering the retrieval performance. To alleviate this issue, we propose CorpusBrain++, a continual generative pre-training framework. Empirical results demonstrate the significant effectiveness and remarkable efficiency of CorpusBrain++ in comparison to both traditional and generative IR methods.

1 INTRODUCTION

Knowledge-intensive language tasks depend on retrieving external knowledge, but generative retrieval research has largely assumed static corpora. This work frames continual document learning for KILTs, evaluates CorpusBrain on KILT++, and introduces CorpusBrain++ to address the challenges of evolving corpora and catastrophic forgetting.

1 INTRODUCTION

  • Prior KILT retrieval models, including CorpusBrain [8], assume stationary knowledge sources, while CorpusBrain’s implicit parameterized index makes incorporating new documents a significant challenge.Traditional dense retrieval can add incremental document representations to an external index without updating the retrieval model itself.Source: §1 · pp. 2–3
  • On KILT++, direct insertion and sequential pre-training variants of CorpusBrain are vulnerable to catastrophic forgetting and inadequate for effective, efficient dynamic retrieval.KILT++ splits the original KILT dataset [37] into sessions simulating continual document additions.Source: §1 · p. 3
  • CorpusBrain++ uses a fixed backbone as long-term retrieval memory and task-specific adapters as short-term memory for learning incremental documents.Each adapter receives a pre-training task tailored to its downstream task.Source: §1 · pp. 3–4
  • To limit forgetting, CorpusBrain++ replays old documents semantically similar to incoming documents and applies the task-specific pre-training tasks to both sets.The framework also generates pseudo-query–docid pairs through task-specific pre-training.Source: §1 · p. 4
  • CorpusBrain++ handles the CDL task effectively and efficiently, mitigates forgetting of previously encountered documents, and shows positive forward knowledge transfer.The paper also reports ablations on individual components and analyzes the effectiveness–efficiency trade-off.Source: §1 · p. 4

2 CONTINUAL DOCUMENT LEARNING FOR KILTS

This section defines continual document learning (CDL) for KILTs, constructs KILT++ to evaluate retrieval as documents arrive, and specifies metrics at dataset, task, and benchmark-wide levels.

2.1 Task formulation

  • CDL assumes sequentially arriving unlabeled document batches and updates the model using new and previous documents so it can retrieve across both old and new content.Evaluation uses held-out queries whose relevant documents belong to the accumulated corpus through a given session.Source: §2.1 · p. 4

2.2 Benchmark construction

  • KILT++ splits the original KILT datasets into sessions; its construction uses a 60% Wikipedia base and four equal-sized incremental document sets.Training pairs are filtered to documents in the base set, and dev queries are iteratively filtered for the accumulated document sets.Source: §2.2 · p. 5

2.3.1 Assessing individual downstream datasets.

  • For each dataset, vertical performance evaluates a model on held-out queries whose relevant documents belong to the corpus accumulated through session i.The metric uses a retrieval measure g(·), such as recall.Source: §2.3.1 · p. 6

2.3.2 Assessing individual downstream tasks.

  • Task-level performance averages vertical performance across the specific datasets belonging to that downstream task.The task metric is defined over the set of datasets for that task.Source: §2.3.2 · p. 6

2.3.3 Assessing all downstream datasets.

  • The benchmark-wide vertical performance summarizes retrieval across all downstream datasets in a session.The section also defines P_t,i to support metrics that track comprehensive retrieval across sessions.Source: §2.3.3 · p. 6
  • Average Performance measures end-of-training performance, Backward Transfer measures new-session learning's effect on earlier sessions, and Forward Transfer measures learning on a new session.These metrics assess all downstream datasets across sessions.Source: §2.3.3 · p. 6

3 ANALYSIS OF CORPUSBRAIN ON CDL

This section evaluates CorpusBrain on the continual document learning task and describes its model and training approach. The naive variants show that directly applying CorpusBrain does not effectively address the task.

3.1 Background

  • CorpusBrain is a pre-trained generative retrieval model reported to achieve state-of-the-art retrieval performance for KILTs.Its transformer encoder-decoder represents queries and generates relevant document identifiers, using Wikipedia page titles as docids.Source: §3.1 · p. 7, §3.1.1 · p. 7

3.1.1 Model architecture.

  • CorpusBrain uses Inner Sentence Selection, Lead Paragraph Selection, and Hyperlink Identifier Prediction to create pseudo-query–docid pairs at different granularities.Their inputs draw on document sentences, leading paragraphs, or anchor-linked sentences and surrounding context.Source: §3.1.2 · p. 7

3.1.2 Pre-training tasks.

  • CorpusBrain initializes from BART and optimizes a standard sequence-to-sequence maximum-likelihood objective on pseudo-query and docid pairs.The pretraining objective is applied to pairs generated by the self-supervised tasks.Source: §3.1.3 · p. 7

3.1.4 Fine-tuning process.

  • CorpusBrain is fine-tuned on training data from five downstream KILT tasks and decodes docids with constrained beam search on a docid prefix tree.The paper reports top performance on a number of downstream tasks on the KILT leaderboard.Source: §3.1.4 · pp. 7–8

3.2 Overall performance

  • Approximately 20% VP drop for Direct versus BM25's 9%, while Sequential drops about 47% in the first incremental session; neither naive variant effectively tackles CDL.Sequential is prone to catastrophic forgetting and performs below BM25 on VP in session 4.Source: Table 2 · p. 8, §3.2 · p. 8
  • The analysis compares direct insertion of new docids without changing backbone parameters with sequential pretraining over documents from the base through each incremental session.The study uses these two naive variants to assess pre-trained CorpusBrain on CDL.Source: §3 · p. 7, §3.2 · p. 8

4 METHODOLOGY

CorpusBrain++ uses task-specific adapters and pre-training objectives to learn incrementally while retaining a shared backbone. The backbone is a transformer encoder-decoder that is initially pre-trained and fine-tuned before its parameters are held fixed.

4.1 Model overview

  • The backbone is initially pre-trained on the base documents through ISS, LPS, and HIP, then fine-tuned using golden pairs from KILT++ training.A dedicated adapter and pre-training task are allocated for each KILT task to reflect its downstream input queries.Source: §4 · p. 8, Figure 3 · p. 9
  • CorpusBrain++ keeps a shared backbone fixed while task-specific adapters learn new documents using objectives tailored to each KILT task.The model also rehearses old documents with the task-specific pre-training objectives to address catastrophic forgetting.Source: §4.1 · p. 9

4.2 Shared backbone

  • The shared backbone is a transformer encoder-decoder with self-attention, cross-attention in the decoder, feed-forward layers, residual connections, and layer normalization.Its parameters are pre-trained on the initial document set and fine-tuned on KILT++ training pairs before being held fixed.Source: §4.2 · pp. 9–10, §4.1 · p. 9

4.3 Task-specific adapter

CorpusBrain++ assigns an independent adapter to each downstream KILT task to capture its query-to-docid mapping. Adapters are placed throughout both encoder and decoder sub-layers.

4.3 Task-specific adapter

  • Each downstream KILT task has its own compact adapter for capturing that task’s query-to-docid mapping.The task-specific adapters are continually pre-trained to accomplish continual document learning.Source: §4.3.1 · p. 10, §4.3 · p. 10

4.3.1 Adapter insertion.

  • Adapters are inserted after sub-layers in both encoder and decoder because tasks differ in query form and query-to-docid mapping.The encoder accommodates variation in query semantic granularity and format; the decoder accommodates variation in mappings.Source: §4.3.1 · pp. 10–11
  • The adapter uses a low-rank layer as its transformation function.The authors describe this structure as simple and effective, citing promising empirical results in prior work [6] [45].Source: §4.3.2 · p. 11

4.4 Task-specific pre-training objective

Each KILT task receives a pre-training objective that constructs pseudo-queries and target docids to resemble its downstream query-document relationships. The constructions vary with query granularity and the number of supporting documents required.

4.4 Task-specific pre-training objective

  • Task-specific pre-training constructs pseudo-queries and output docids to mimic each downstream task’s query-document relationship.Pseudo-query construction varies by semantic granularity, while the number of target docids reflects the task’s output scenario.Source: §4.4 · pp. 11–12

4.4.1 Fact checking.

  • Fact-checking pseudo-queries combine a sampled sentence span with the document title, while targets can include the source document and linked pages.The method samples anchor texts and concatenates the source docid with docids of their destination pages.Source: §4.4.1 · p. 12

4.4.2 Entity linking.

  • Entity-linking pseudo-queries use a sentence containing a sampled anchor text marked with entity-boundary tokens, and target its linked destination page.Only one target docid is required for this task.Source: §4.4.2 · pp. 12–13

4.4.3 Slot filling.

  • Slot-filling pseudo-queries pair a document title with predicted relation types from sampled sentences, with the document itself as the target.A BERT relation detector trained on the initial-session slot-filling data supplies the top-k relation types.Source: §4.4.3 · pp. 13–14

4.4.4 Open-domain QA.

  • Open-domain QA pseudo-queries turn sampled sentence spans into short questions that include the document title, and targets may include linked supporting pages.The interrogative word is selected from a predefined set, and the output concatenates the source docid with linked-page docids.Source: §4.4.4 · p. 14

4.4.5 Dialogue.

  • Dialogue pseudo-queries combine sampled leading paragraphs with constructed questions, while targets can include linked pages supporting the conversation.The questions are appended to, rather than substituted for, the paragraph-level context.Source: §4.4.5 · pp. 14–15

4.5 Learning process

CorpusBrain++ learns each incoming document session by updating task-specific adapters on pseudo-query/docid pairs. It rehearses semantically similar documents from earlier sessions and optimizes a sequence-to-sequence objective over both new and rehearsed data.

4.5.1 Learning new documents.

  • For each new document session, task-specific pre-training tasks generate pseudo-query/docid pairs that train the corresponding task adapter.The new-document learning process updates adapters rather than the shared backbone.Source: §4.5.1 · p. 15, §4.5.3 · p. 16

4.5.2 Rehearsing old documents.

  • To rehearse old documents, the method clusters previous-session docids with K-means and samples old documents from clusters matching each new document.The resulting experience set is used to construct additional pseudo-query/docid pairs for adapter training.Source: §4.5.2 · pp. 15–16
  • At each session update, each task adapter is initialized from its previous-session parameters, while the shared backbone remains unchanged.The first incremental session instead randomly initializes the adapters.Source: §4.5.3 · p. 16

4.5.3 Overall learning objective.

  • The overall objective sums task-specific sequence-to-sequence learning over pseudo-query/docid pairs derived from new and rehearsed old documents.Each task’s transformation function constructs its pairs, and its adapter parameters are updated across the session.Source: §4.5.3 · p. 16

4.6 Inference process

At inference, CorpusBrain++ groups test queries by task and uses the shared backbone with the corresponding task adapter to generate document titles. Constrained beam search limits outputs to titles of documents available by the current session.

4.6 Inference process

  • Test queries are grouped by task, and each group uses the shared backbone with its task-specific adapter to generate document titles.This combines task-shared and task-specific knowledge.Source: §4.6 · p. 16
  • Constrained beam search restricts generated docids to a predefined set of document titles.The prefix tree contains only titles for documents available through the current session.Source: §4.6 · p. 16
  • Each prefix-tree node represents a token, and a path from the root forms a document title.Source: §4.6 · p. 16

5 EXPERIMENTAL SETTINGS

The experiments compare CorpusBrain++ with traditional and generative retrieval models, test architectural, objective, and rehearsal variants, and report retrieval using R-precision.

5.1 Models

  • The comparison includes CorpusBrain++, traditional IR models, and generative IR models.BM25 and DPR are traditional baselines; generative baselines include stationary and dynamic-scenario models.Source: §5.1.1 · pp. 16–17

5.1.1 Baselines.

  • GENRE and SEAL are compared only in the non-incremental setting, while DSI++ and CLEVER represent generative baselines for dynamic retrieval.In the non-incremental setting, CorpusBrain++ reduces to CorpusBrain, whose comparisons with GENRE and SEAL were previously reported.Source: §5.1.1 · p. 17

5.1.2 Ablation models.

  • Ablations vary backbone versus adapter updates, original versus task-specific pre-training objectives, and semantic-similarity, random, or no document rehearsal.The backbone variants use either single-task or multi-task continual pre-training.Source: §5.1.2 · p. 17

5.2 Implementation details

  • The implementation uses Wikipedia titles as docids, BART-large, and an initial phase followed by incremental learning sessions.Initial pre-training uses ISS, LPS, and HIP; incremental sessions pre-train task-specific adapters using new and revisited documents.Source: §5.2 · pp. 17–18

5.3 Evaluation metrics

  • Retrieval on the KILT++ test set is evaluated with R-precision, the metric recommended by KILT’s official instructions.R-precision uses the number of relevant documents among the top R retrieved pages, where R is the size of the golden provenance set.Source: §5.3 · p. 18

6 EXPERIMENTAL RESULTS

The experiments assess incremental retrieval across KILT datasets and tasks, forgetting, forward transfer, efficiency trade-offs, and example retrieval behavior. CorpusBrain++ leads across the reported overall metrics, while its performance varies with task and design choices.

6.1.1 Incremental performance of individual downstream datasets.

  • On individual datasets, CorpusBrain++ surpasses traditional and generative methods on most datasets, but trails DPR on NQ and is unstable on long, complex-query datasets.The authors identify ELI5 as especially challenging because it has no training data; DSI++ and CLEVER do not sustain their retrieval ability incrementally.Source: Table 3 · p. 19, §6.1.1 · p. 20

6.1.2 Incremental performance of individual downstream tasks.

  • Across tasks, gains are stronger for entity-centric fact checking and entity linking, modest for slot filling, and unstable for open-domain QA and dialogue.The authors associate the latter variability with queries that may not contain entities from the target document, making task behavior harder to simulate in pre-training.Source: §6.1.2 · pp. 20–21, Figure 6 · p. 21

6.1.3 Incremental performance of all datasets.

  • CorpusBrain++ achieves the best overall performance across VP, AP, BWT, and FWT, outperforming traditional and generative retrieval methods.Table 4 reports comprehensive performance across datasets; the authors conclude the framework adapts effectively to dynamic retrieval.Source: Table 4 · p. 21, §6.1.3 · p. 22
  • Direct and Sequential lose retrieval performance as documents arrive, while DSI++ and CLEVER nearly lose retrieval capability during incremental learning.The authors attribute generative-baseline difficulty to heterogeneous KILT queries and to their docid choices; the Direct and Sequential results indicate forgetting in off-the-shelf CorpusBrain.Source: §6.1.3 · p. 22

6.2 Assessing catastrophic forgetting

  • CorpusBrain++ nearly prevents forgetting; task-specific pre-training and semantic-similarity-based rehearsal outperform general objectives and random rehearsal in the reported comparisons.The no-rehearsal variant has the poorest BWT, and the backbone-adapter variants outperform backbone-only variants across metrics.Source: §6.1.3 · pp. 22–23, Figure 7 · p. 23, §6.2 · p. 24

6.3 Assessing forward transfer

  • From session 2 onward, CorpusBrain++ substantially outperforms Individual, supporting forward transfer from prior sessions.The two methods’ training is equivalent in sessions 0 and 1; Table 5 presents the forward-transfer comparison.Source: Table 5 · p. 24, §6.3 · p. 24

6.4 Effectiveness-efficiency trade-off

  • CorpusBrain++ pairs superior retrieval performance with only marginally higher memory and training-time overhead than Direct.The plotted memory and training-time ratios are relative to Direct; Sequential incurs substantial training time without satisfactory retrieval performance.Source: §6.4 · pp. 24–25, Figure 8 · p. 25

6.5 Case study

  • In the case study, CorpusBrain++ learns the new Nelson Mandela document in session 1 and retains its retrieval capability across sessions.The examples include entity linking and question answering; the QA example retrieves both a related document and the newly learned document.Source: §6.5 · p. 25

7 RELATED WORK

Related work covers KILTs, generative retrieval, and continual learning. The paper situates its contribution in dynamic KILT retrieval, where prior KILT research largely assumed a static corpus.

7.1 Knowledge-intensive language tasks

  • KILTs use external knowledge, and KILT benchmarks eleven datasets across five tasks grounded in one Wikipedia snapshot.Common retrieval pipelines select relevant information for a reader to use in generating a response.Source: §7.1 · p. 26

7.3 Continual learning

  • Unlike prior KILT work focused on static corpora, this paper studies continual document learning as the knowledge corpus evolves without corresponding KILT queries.The approach combines parameter isolation with replay methods for continual learning.Source: §7.1 · p. 27, §7.3 · p. 28

7.2 Generative retrieval

  • Generative retrieval replaces multi-stage indexing and ranking with a model that stores document information in its parameters and generates relevant docids.Prior work studies docid representations and query-to-docid mappings; CorpusBrain uses pre-training tasks to generate pseudo-queries.Source: §7.2 · p. 27, §7.1 · p. 27

8 CONCLUSION AND FUTURE WORK

The paper introduces continual document learning (CDL) and KILT++ to assess dynamic retrieval for KILTs, and presents CorpusBrain++ for this setting. It reports an effective and efficient framework while identifying limits in the continual-learning paradigms studied and future directions involving large language models.

8 CONCLUSION AND FUTURE WORK

  • The paper defines continual document learning (CDL) for KILTs and introduces KILT++ to systematically assess dynamic retrieval.The framework is presented as addressing the CDL task.Source: §8 · p. 28
  • CorpusBrain++ combines a backbone-adapter architecture with task-specific pre-training objectives and semantic-similarity document rehearsal to support continual learning and address catastrophic forgetting.The objectives are tailored to downstream KILT tasks; the paper reports the framework as effective and efficient.Source: §8 · p. 29
  • The work explores generative retrieval in dynamic scenarios across multiple, diverse downstream tasks and proposes its benchmark as a basis for further research.The authors describe this as the first such exploration to their knowledge.Source: §8 · p. 29
  • The authors identify their focus on parameter isolation and experience replay as a limitation, and suggest studying regularization-based and additional parameter-isolation methods.They also point to task-specific pre-training objectives designed in collaboration with large language models as future work.Source: §8 · p. 29

REPRODUCIBILITY

The paper reports using open datasets for reproducibility and makes its code and constructed benchmark data available.

REPRODUCIBILITY

Loading 2402.16767v1…