Source-linked AI summary
RealTime QA: What's the Answer Right Now?
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, Kentaro Inui
TL;DR
REALTIME QA addresses the mismatch between static QA benchmarks and users’ need for answers about newly changing information. It introduces a weekly dynamic benchmark with open-book and closed-book baselines, finding that current retrieval substantially improves results but retrieval gaps still produce outdated answers.
Problem
REALTIME QA addresses the limitation of static open-domain QA benchmarks for questions whose answers change with current events and information.
Method
The paper builds a weekly dynamic benchmark with continually evaluated open-book and closed-book baselines based on pretrained models and retrieved news evidence.
Results
Open-book GPT-3 with Google custom search substantially outperforms closed-book baselines and retrieval from a past Wikipedia dump, while retrieval errors remain common.
Takeaways & Limitations
REALTIME QA highlights accurate, up-to-date information retrieval as particularly important for serving instantaneous QA and broader real-time NLP applications.
Takeaways & Limitations
The current benchmark is English- and US/UK-centric and announces questions weekly rather than on a truly instantaneous basis.
Abstract
from arXiv · showhide
We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). REALTIME QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that REALTIME QA will spur progress in instantaneous applications of question answering and beyond.
1 Introduction
REALTIME QA extends time-sensitive QA toward questions about the current world by providing a dynamic benchmark and weekly evaluation platform. Its baseline results show that up-to-date retrieval improves performance, while retrieval failures remain a central challenge.
- Motivation: REALTIME QA benchmarks questions about current, changing information rather than assuming that answers remain static over time.It extends prior time-sensitive QA work toward more instantaneous information needs.
- Benchmark design: The platform announces human-written questions and evaluates submissions weekly across politics, business, sports, and entertainment.Questions and baseline results are uploaded each week, with evaluation open until the next question set.
- Baselines: The study compares open-book and closed-book real-time QA baselines built on pretrained models including GPT-3, T5, and BART.Open-book models use external knowledge sources, whereas closed-book models answer without access to newly available information.
- Data and evaluation: 1,470 QA pairs support the real-time evaluations, alongside 2,886 earlier pairs released for future model development.The reported evaluation period runs from June 17, 2022 through June 2, 2023.
- Findings: Open-book GPT-3 with up-to-date retrieval substantially outperforms closed-book baselines and open-book models retrieving from a past Wikipedia dump.The results indicate that GPT-3 can adjust its answers using retrieved passages.
- Findings: Despite these gains, models struggle when choices express uncertainty, and most errors originate from retrieval rather than reading comprehension.The benchmark therefore emphasizes fast, accurate retrieval for instantaneous QA applications.
2 REALTIME QA Framework
REALTIME QA combines weekly question annotation, web retrieval, optional shared evidence, and continuous baseline evaluation. It supports multiple-choice and generation settings while contrasting systems that can update external knowledge with closed-book systems constrained by training data.
- Workflow: Questions are announced weekly from recent news, with approximately 30 multiple-choice questions posted at 3 am GMT every Saturday.The workflow uses news-based quizzes and evaluates systems before the next question set is announced.
- Question annotation: Questions are extracted from weekly quizzes on CNN, USA Today, and The WEEK, covering genres such as politics and business.The current version excludes quizzes from some other available sources.
- Document retrieval: Google custom search retrieves the top-10 web documents for each question, which are parsed and stored with text and metadata.Participants may use the shared document set or create their own knowledge sources.
- Evaluation: Multiple-choice performance is measured by accuracy, while generation uses exact matching and token-based F1 scores.The benchmark also evaluates a NOTA setting in which one choice is replaced with “none of the above.”
- Model classes: Open-book systems retrieve external evidence before answer prediction, whereas closed-book systems answer directly without external knowledge.Closed-book models cannot access information newer than their pretraining or finetuning data.
- Prompting: The GPT-3 open-book prompt prepends retrieved article titles, opening paragraphs, and publication dates to each question.The closed-book prompt omits the articles, and date insertion is evaluated through ablation studies.
3 Experiments and Analysis
REALTIME QA evaluates real-time QA baselines over a year, showing that current retrieval substantially improves performance while temporal gaps and uncertainty remain difficult.
- GPT-3 with Google custom search achieved the best performance across original multiple choice, NOTA multiple choice, and generation settings.
- 34.6 vs. 15.3/13.3 in generation exact matching shows GPT-3 with GCS outperforming closed-book GPT-3 and GPT-3 with DPR.
- 58.4 vs. 66.5 for GPT-3 with GCS retrieval shows a large performance drop from original multiple choice to NOTA.
- Temporal specification almost always helps open-book GPT-3 but hurts the closed-book model, possibly because dates are unseen during pretraining.
- 52% of answers were correct, while 34% involved retrieval errors and 13% involved reading-comprehension errors.
- Performance remains high or improves up to around 24 hours after announcement, then substantially degrades as temporal gaps emerge.
- When retrieved documents do not suffice, GPT-3 can ignore temporal gaps and generate outdated answers.
4 Related Work
REALTIME QA connects dynamic QA to temporal retrieval, summarization, temporal misalignment, and evolving benchmark design while emphasizing evaluation with changing information.
- Temporal summarization retrieves web documents and summarizes them over time, but its evaluations are usually static.
- TREC real-time summarization evaluates systems during a 1–2 week period, alongside work on financial news and emergency-management applications.
- Table 3 illustrates both successful answer updates from newly retrieved documents and outdated answers when retrieval is outdated or unrelated.
- REALTIME QA addresses temporal misalignment concerns alongside prior work studying performance degradation between training and evaluation or deployment.
- Unlike most NLP datasets, REALTIME QA evaluates systems dynamically as its test data and evaluations change over time.
- Open-domain QA has increasingly challenged fixed-knowledge setups by incorporating temporal or geographical contexts that can change answers.
5 Conclusion and Future Work
The paper concludes that REALTIME QA is a dynamic, weekly benchmark for present-time open-domain QA and that accurate, current retrieval is especially important.
- REALTIME QA asks present-time questions, announces them weekly, and continually evaluates six real-time baselines.
- Results from the first year suggest that accurate, up-to-date retrieval is particularly important for serving speedy information needs.
- The authors hope the benchmark encourages research toward fast, accurate NLP applications.
Limitations
REALTIME QA is limited by English- and US/UK-centric coverage and weekly rather than truly instantaneous question announcements.
- Question-answer pairs are written in English, and covered topics tend to be English-centric, particularly involving the US and UK.
- Questions are announced weekly rather than on a truly instantaneous basis.
- Future versions are intended to mitigate these annotation-framework limitations.
Checklist
The paper’s checklist reports affirmative coverage of contributions, limitations, societal impacts, ethics, reproducibility, training details, compute, and asset licensing.
- The authors report that they described the paper’s contributions, scope, limitations, societal impacts, and ethics compliance.
- The checklist reports that theoretical-results questions were not applicable.
- The authors report providing code, data, and reproduction instructions for the main experiments.
- Training details and resource information were reported, while error bars were marked not applicable.
- The authors report citing existing assets and mentioning their licenses.
- The checklist records affirmative or not-applicable responses concerning participant consent, personally identifiable information, offensive content, participant risks, and compensation.
A Baseline Configurations
The paper provides reproducible baseline configurations, including dense passage retrieval and retrieval-augmented generation settings from the Transformers library.
- The authors generally follow the Transformers library’s default settings for these baseline configurations.
- The configurations cover dense passage retrieval and retrieval-augmented generation from the Transformers library.
B REALTIME QA Interface
The REALTIME QA interface updates weekly, evaluates six baselines when questions become available, and displays submissions with their submission times.
- The interface is updated every week and evaluates all six baselines as soon as questions are available.
- The same page displays submissions together with their submission times.
C REALTIME QA Statistics
REALTIME QA statistics summarize the first four weeks by genres and answer types, including a minority of questions that were not strictly time-sensitive.
- The first four weeks are analyzed by question genres and answer types.
- ∼10% of questions were not strictly time-sensitive in the reported first-four-week statistics.The paper retains these cases to simulate naturally occurring information-seeking scenarios.
- Table 6 reports detailed REALTIME QA statistics as percentages.