StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

1. Introduction

• Existing QA datasets lack the question-date and document-date grounding needed to evaluate adaptation to evolving knowledge while retaining answers about the past.Prior work also used static knowledge snapshots or added question dates while still answering over static Wikipedia.Source: §1 · pp. 1–2
• StreamingQA pairs dated questions with a streaming corpus of 14 years of time-stamped English WMT news articles and includes human-written and generated questions.Its temporal metadata enables periodic knowledge ingestion and evaluation on questions asked during that period.Source: §1 · p. 2
• The study evaluates adaptation in parametric closed-book and semi-parametric open-book QA, including questions about recent and past knowledge.It also establishes benchmarks for one-step streaming and the usual static open-book QA task.Source: §1 · p. 2
• Parametric adaptation improves open-book QA, while parametric updates particularly help frequent-entity questions and retrieval-based adaptation helps less-frequent knowledge.Incremental fine-tuning works reasonably well without catastrophic forgetting and uses 95% fewer steps per month than full retraining in this setup.Source: §1 · p. 2

2. StreamingQA Dataset and Task

• The task supplies question dates and article publication dates so models can be evaluated on knowledge that becomes available incrementally, with training and evaluation drawn from non-overlapping historical periods.The dataset uses 14 years (2007–2020) of English WMT news articles, with questions generated or written by annotators and then filtered.Source: §2 · p. 2

2.1. StreamingQA Task: QA with Temporal Metadata

• For each period, questions asked during that period are answered from the corpus published by its end; evaluation quarters are split into recent and past questions.Recent questions concern articles from the preceding month, while past questions concern articles published from 2007 up to the question date.Source: §2 · pp. 2–3, §2.1 · p. 3

2.2. Questions Dates

• Evaluation questions use recent articles dated Dec 2019–Dec 2020, with dates from publication through 30 days later, and past articles dated 2007–Dec 2020, with dates from max(publication, Jan 2020) through 365 days later.Source: §2.2 · p. 3

2.3. Automatically Generated Questions

• Generated questions use few-shot prompting of a large language model with an evidence document and target answer, followed by filters for trivial or low-quality candidates.Past-subset questions receive absolute or relative time specifications unless the question already contains one or the answer is a date.Source: §2.3 · p. 3

2.4. Human Written Questions

• Human annotators wrote questions and answers about provided news articles, with instructions to include enough context for unambiguous open-book questions.Annotators were asked to write up to five questions per article.Source: §2.4 · pp. 3–4

2.5. Quality Filtering

• Quality filtering required annotators to judge questions factual, unambiguous, and grammatical, then answer them using the original passage and dates.Three annotators had to agree to include a question; only answerable questions were kept, with additional reference answers collected.Source: §2.5 · p. 4

2.6. Statistics

• The evaluation set contains about 28k generated and 8.8k human-written questions, with 100k training and 10k validation questions.For written evaluation questions, answer types are about 46.9% named entities, 40.6% phrases, and 12.4% dates; about 6% of evaluation reference answers occur among training answers, while 23% of training questions have an answer among evaluation references.Source: §2.6 · p. 4

3. Experimental Setup

• The study evaluates Transformer-XL as a closed-book model and as the language model underlying a RAG-style open-book model, plus Fusion-in-Decoder with T5.F1 and exact match are used after answer normalization.Source: §3 · p. 4

3.1. Language Model Pretraining and Finetuning

• Transformer-XL models are trained on articles through 2019, retrained through 2020, or incrementally fine-tuned on monthly 2020 article sets.The stale and retrained models use 200k training steps; monthly fine-tuning uses 10k steps, and T5 is also fine-tuned on the article splits.Source: §3.1 · pp. 4–5

3.2. Closed-book QA

• Closed-book models answer without additional context and are evaluated on 2020 questions after QA fine-tuning on questions and knowledge from 2019 or earlier.The best checkpoint is selected using validation F1, and answers are produced by greedy decoding.Source: §3.2 · p. 5

3.3. Open-book QA

• Open-book QA retrieves passages from dated WMT news articles, split into six-sentence chunks, and uses RAG-style or Fusion-in-Decoder models.The RAG-style model retrieves 20 passages; Fusion-in-Decoder retrieves 64 and attends to all retrieved passages together. Training uses a restricted search space containing gold evidence for training and validation questions.Source: §3.3 · p. 5

4.1. Adaptation to New Knowledge and Forgetting in Closed-Book QA

• CB+RETR. outperforms or matches all CB+FT models, which improve over CB+STALE as new monthly documents are added.The improvement is driven by better recent-subset accuracy while past-subset performance remains mostly unchanged.Source: §4.1 · p. 5
• As fine-tuning lag grows, past-subset performance falls to about 5% below CB+RETR., while recent-subset performance first rises and then shows minor forgetting.Similar conclusions hold for written questions.Source: §4.1 · p. 6

4.2. Adaptation to New Knowledge and Forgetting in Open-Book QA

• On recent generated and written questions, open-book models adapt steeply from -1Q to 0Q, even with index updates alone, and show almost no forgetting; generated past questions show no forgetting.A small fraction of past questions reference 2020 articles, so seeing 2020 knowledge slightly helps compared with a lagging model.Source: §4.2 · p. 6
• Generator updates improve recent-subset performance beyond index updates, and retraining the generator improves all models and subsets.For FID/T5, fine-tuning performs better than retraining on recent generated questions but worse on the past subset.Source: §4.2 · pp. 6–7

4.3. Parametric vs Semi-parametric Adaptation

• For frequent named entities in questions, closed-book F1 is higher by absolute 10%; at higher frequencies parametric adaptation helps, while at lower frequencies updated search space drives better performance.DPR recall@1 decreases by absolute 5% as question-entity frequency increases, offset by improved generator performance.Source: §4.3 · p. 7

4.4. Further Analyses

• DPR with dates retrieves gold articles at a median temporal difference of 41 days versus 1101 days without dates, and recent generated-question recall@20 is 57% versus 43%.Past questions show no improvement; the authors suggest models may misinterpret prepended question dates alongside time expressions in question text.Source: §4.4 · p. 7
• Open-book models perform best on generated past questions without time expressions: FID+IU+FT F1 is 0.711 without, 0.469 with absolute, and 0.359 with relative specifications.For FID+IU+FT on past generated questions, likely static questions score 0.494 F1 versus 0.435 for updated questions; likely static questions are 6.7%–11.2% of evaluation sets.Source: §4.4 · pp. 7–8

4.5. One-step Streaming and Static QA Benchmarks

• The one-step streaming and static open-book benchmarks leave a large gap to human performance and expose challenges in retrieval and news-reading comprehension.The comparison includes models given gold evidence versus retrieved evidence.Source: §4.5 · p. 8, Figure 8 · p. 8

• Prior QA work focused on passages, books, static corpora, or temporally grounded facts without question dates, while this work evaluates adaptation in downstream QA.The passage contrasts prior temporal work on knowledge graphs and news with the absence of question-date grounding.Source: §5 · p. 8
• Temporal language-model studies documented post-training deterioration and adaptation strategies, while this work connects temporal degradation to downstream QA performance.The related work cites Lazaridou et al. (2021) and Röttger & Pierrehumbert (2021) on temporal adaptation in language modeling and classification.Source: §5 · p. 8
• Semi-parametric approaches offer flexible adaptation through retrieval updates, and prior open-domain QA research studied retrieval, multi-document combination, and named entities.The cited work includes Khandelwal et al. (2020), Lewis et al. (2020a), Sachan et al. (2021), Izacard & Grave (2020), Sciavolino et al. (2021), and Liu et al. (2021).Source: §5 · p. 8

• StreamingQA provides a dataset and task for studying adaptation to new information over time in open- and closed-book QA, with temporally non-overlapping training and evaluation sets.Source: §6 · p. 8
• Open-book models adapt quickly through search-space updates, while fine-tuning or retraining generally further improves performance.The ability to inject knowledge through retrieval depends on retrieval accuracy, and more up-to-date LMs can compensate for retrieval errors.Source: §6 · pp. 8–9
• Iterative fine-tuning improves the FiD QA model, so costly retraining from scratch is not necessary in the studied setup.Closing the performance gap between retrained and stale generators remains future work.Source: §6 · p. 9

• News-based answers follow the articles regardless of their factual basis, and the dataset may not apply to settings where its assumptions do not hold.The authors note that news can contain graphic descriptions, quotes, or opinions that may be considered toxic.Source: §7 · p. 9
• Perspective API category classifiers were thresholded using manual review; the authors caution that false positives could reduce representation of some groups.Two annotators judged samples of 50 questions in score bands, and the team annotated more than 5.5k examples during threshold setting.Source: §7 · p. 9
• Filtering removed about 0.57% of questions overall, and the authors estimated that 0.5% of toxic questions remained.Removal rates were 0.60% for Train, 0.61% for Valid, 0.43% for Eval-Generated, and 0.65% for Eval-Written.Source: §7 · p. 9

• Table 4 presents a comparison of StreamingQA with other related datasets.Source: §A.1 · p. 13
• The comparison’s stated scope is StreamingQA and related datasets.Source: §A.1 · p. 13
• The passage identifies Table 4 as the location of the dataset comparison.Source: §A.1 · p. 13

A.2. Answer types in evaluation set

• The evaluation-set answer types are detailed in Figure 9, while Figure 10 shows question and article publication-date distributions for Eval-Written.The Eval-Generated date distribution is described as similar.Source: §A.2 · p. 13, Figure 10 · p. 14

A.3. Examples of questions with high- and low-frequency named entities

• Table 5 provides examples of questions with high- and low-frequency named entities.Source: Table 5 · p. 13, §A.3 · p. 14

A.4. Automatic filtering of trivial and/or low-quality generated questions

• Generated questions are filtered for answer leakage, answer recoverability, search-result support, and named-entity presence in phrase-answer questions.The filters use few-shot LM QA and Google Search against the evidence publication date.Source: §A.4 · p. 14

A.5. Adding time specification to past generated questions

• Past generated questions receive a randomly selected absolute month-and-year or relative-time specification; Figure 11 reports whole-dataset F1 by answer type across training cut-off dates.Source: §A.5 · p. 14, Figure 11 · p. 15

A.6. Dataset Toxicity Filtering

• The authors adapted Perspective API classifier definitions for manual annotation used to set filtering thresholds.Source: §A.6 · p. 15
• The adapted toxicity categories cover identity attack, insult, profanity, severe toxicity, sexually explicit content, and threats.Source: §A.6 · p. 15

B.1. Closed-book

• Closed-book analysis examines written-question F1 by temporal lag and compares adaptation on recent articles with forgetting on past articles.Figure 13 contrasts fine-tuned and stale models for adaptation, and fine-tuned and retrained models for forgetting.Source: §B.1 · p. 15, Figure 12 · p. 16, Figure 13 · p. 16

B.2. Open-book

• T5 masked-span results show forgetting on past evaluation documents and slight recency bias on recent documents relative to retraining.The open-book analysis also presents past-written-question comparisons with closed-book models and metrics for all subsets.Source: Figure 15 · p. 17, §B.2 · p. 17, Figure 14 · p. 17

B.3. Temporal accuracy of DPR trained on news

• For Q4’2020 recent questions, DPR with timestamp more closely matched the gold passages’ temporal distribution, though its second lower spike fell in Q4’2019.Source: §B.3 · p. 17

B.4. One-step Streaming and Static QA Benchmarks

• The static setup reports EM in Figure 17 and all metrics in Table 7; filled bars mark models incrementally given 2020 knowledge.Source: §B.4 · p. 17, Figure 17 · p. 18, Table 7 · p. 19
• DPR with timestamp matches the gold passages’ temporal distribution more closely, though its second lower spike falls in Q4’2019.The comparison concerns recent Q4’2020 questions.Source: Figure 16 · p. 18
• Table 6 presents adaptation and forgetting F1 scores with 95% confidence intervals.Source: Table 6 · p. 19