PaperlayerView exact revision on arXiv

Source-linked AI summary

SituatedQA: Incorporating Extra-Linguistic Contexts into QA

Michael J. Q. Zhang, Eunsol Choi

arXiv:2109.06157v1cs.CL

TL;DR

SituatedQA addresses the problem that answers to information-seeking questions can depend on temporal or geographical context, while standard QA evaluation often treats them as fixed. It constructs a context-aware open-retrieval dataset and evaluates adaptation across situations, finding substantial failures on updated facts and uncommon locations. The paper argues that QA benchmarks should incorporate extra-linguistic context to remain relevant globally and over time.

  • Problem

    Answers to the same question may change with when and where it is asked, but existing QA evaluation does not measure adaptation to such contexts.

  • Method

    SituatedQA re-annotates questions from existing QA datasets for temporal or geographical dependence and collects answers for alternative contexts.

  • Results

    15 percentage points lower accuracy occurs on updated-answer questions, while accuracy drops 10 percentage points for rare versus common locations.

  • Takeaways & Limitations

    QA benchmarks should incorporate extra-linguistic contexts to support systems that adapt to changing temporal and geographical facts.

  • Takeaways & Limitations

    Temporal and geographical contexts are not modeled jointly for simplicity.

Abstract

from arXiv · show

Answers to the same question may change depending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SituatedQA, we first identify such questions in existing QA datasets. We find that a significant proportion of information seeking questions have context-dependent answers (e.g., roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their corresponding answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering questions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should incorporate extra-linguistic context to stay relevant globally and in the future. Our data, code, and datasheet are available at https://situatedqa.github.io/ .

1 Introduction

SituatedQA studies open-retrieval questions whose answers vary with temporal or geographical context, addressing evaluation assumptions that treat questions as context-independent. It introduces context-aware tasks, annotations, and analyses showing that current systems struggle with changed facts and unfamiliar locations.

  • Answers to some open-retrieval questions vary with when or where the question is asked.
  • Existing QA evaluation implicitly assumes fixed time and location and does not measure adaptation to new contexts.
  • SituatedQA re-annotates questions from existing open-retrieval datasets for temporal or geographical dependence and collects answers across situations.
  • 9K questions from four datasets receive temporal annotations, while 2K receive geographical annotations; annotators find up to 30% can change over time.
  • 15 percentage points lower accuracy occurs on updated-answer questions, and accuracy drops 10 percentage points for rare versus common locations.
  • The paper proposes tasks and fine-grained evaluations for modeling what changes across contexts and how systems adapt to new temporal and geographical settings.

2 Definitions & Tasks

The paper defines temporal and geographical contexts and formulates tasks for answering questions within a supplied context and identifying context-dependent questions. These tasks also support evaluation of generalization to new contexts.

  • Defining Extra-Linguistic Contexts: A context consists of a type and value, with the paper studying temporal and geographical types.
  • Defining Extra-Linguistic Contexts: Temporal context values are timestamps at which an answer applies, while geographical values are geopolitical entities where the answer applies.
  • Situated Question Answering: The situated question-answering task maps a question and supplied context to its corresponding answer.
  • Situated Question Answering: Models are evaluated by exact string match after minor normalization between annotated and predicted answers.
  • Situated Question Answering: Evaluating different context sets measures generalization across common and rare locations or recently and long-ago changed answers.
  • Context-Dependent Question Identification: Context-dependent question identification is binary classification of whether distinct context values yield different answers, evaluated with accuracy, F1, precision, and recall.
  • Context-Dependent Question Identification: Identifying changing facts can require extensive background knowledge, such as knowing whether Kazakhstan’s capital has changed.

3 Data Collection

The authors construct SituatedQA by identifying temporally and geographically context-dependent questions, collecting alternate context–answer pairs, and validating annotations. The resulting dataset captures changing answers, geographic variation, and annotation agreement across multiple QA sources.

  • Collection pipeline: Data collection uses three stages: identifying context-dependent questions, collecting context–answer pairs, and validating responses.For temporal questions, workers provide answer timelines; for geographic questions, they provide location–answer pairs.
  • Source datasets: The source questions come from Natural Questions, WebQuestions, TyDi-QA, and MS-MARCO, whose English questions are answerable from Wikipedia.
  • Geographic collection: Geographically dependent questions are augmented by modifying existing NQ-Open questions because geographic ambiguity often leads to their rejection during annotation.
  • Temporal collection: Temporal examples derive from worker-provided current and previous answers with transition timestamps, instantiated as Start, Sampled, or Static context values.Start uses an answer’s transition timestamp, while Sampled uses timestamps between transitions; Static examples represent temporally independent questions.
  • Quality control: Validation uses another annotator for development and test questions, while training timelines receive a single annotation without validation.Agreement is moderate to high overall but lower for ambiguous questions and questions requiring extensive domain knowledge.
  • Dataset construction: For 2.8K context-dependent questions, workers supplied 5.9K alternate-context answers, yielding 6K temporal context examples alongside 6.7K temporally independent examples.
  • Temporal analysis: Answers change around the one-year mark for a large proportion of temporally dependent questions, producing a long-tailed distribution of previous-answer durations.

4 Models

The study evaluates retrieval-based and closed-book QA baselines under increasingly context-aware settings. It compares unmodified models with query modification and context-specific fine-tuning, using updated retrieval evidence for DPR.

  • Baselines: The experiments include retrieval-based and closed-book QA models, represented by DPR and BART-large, respectively.
  • Baselines: Both baselines are trained on NQ-Open, while DPR retrieves from an English Wikipedia dump dated 2021-Feb-20.In standard NQ-Open settings, DPR reaches 41.5 accuracy and BART reaches 24.1 accuracy.
  • Evaluation settings: Human performance is approximated using author-annotated samples, with discrepancies often arising from ambiguous questions or equivalent answer forms.
  • Evaluation settings: The unmodified setting gives models only the question, requiring one answer regardless of temporal context or an assumed geographic context.
  • Evaluation settings: Query modification appends a context phrase such as “as of cv” or “in cv” to transform situated questions into context-specified inputs.
  • Evaluation settings: Context-specific fine-tuning separately adapts DPR readers and retrievers, and trains separate closed-book models for each context type.

5 Results

SituatedQA evaluations show substantial gaps in context-dependent QA, with retrieval-based systems generally outperforming closed-book models and adaptation helping unevenly across temporal and geographic settings.

  • Context-dependent question identification: Pretrained language models match human agreements when identifying context-dependent questions in binary classification.This suggests they may help identify questions needing re-annotation or unsuitable for closed-book systems.
  • Situated question answering: 41.5 EM vs. 24.1 EM: retrieval-based models outperform closed-book models overall on situated QA.The comparison follows the same pattern observed in the original QA datasets.
  • Situated question answering: Finetuning on answers from multiple contexts improves performance across the board, especially for TEMP questions.The improvement is reported for models trained with modified questions and multiple extra-linguistic contexts.
  • Situated question answering: Adding TEMP context to static questions decreases performance, especially for the closed-book baseline, while answer start dates outperform sampled dates.The findings indicate sensitivity to semantically equivalent edits and to how temporal context is supplied.
  • Geographic context: 10 percentage points vs. 3 points: rare-location performance drops more for closed-book than retrieval-based models.The gap between common and rare locations shrinks after finetuning for both model types.
  • Error analysis: Models often produce an answer from another context rather than the answer associated with the specified context.This error analysis evaluates exact matches against the correct context and against the union of annotated contexts.

6 Analysis

The analysis examines whether models can retrieve previous answers and adapt outdated knowledge to present contexts. Finetuning improves historical-answer queries, while updated retrieval corpora alone are insufficient for updated facts.

  • Current and previous answers: Finetuned baselines use separate reader and closed-book models to query for current and previous answers.The previous-answer query is formed by prepending “previously” in query-modified baselines.
  • Current and previous answers: Models are far better at producing current answers than past answers to the same question.Finetuning greatly increases performance on queries for previous answers.
  • Adapting to the present: The present-answer evaluation splits questions into Stable and Updated categories based on whether answers changed after 2018.Current answers are up-to-date as of February 2021, with 2018 marking the Natural Questions collection period.
  • Adapting to the present: Significant gains on Updated questions appear only after finetuning with newer data, not merely after swapping the retrieval corpus.Retrieval-based models update some world knowledge after corpus replacement, but corpus updates alone are not sufficient.

7 Related Work

SituatedQA extends prior work on ambiguity, temporal generation, temporal understanding, dynamic evaluation, and evolving knowledge sources by studying context-dependent open-retrieval QA with manually annotated examples.

  • Comparison to other QA datasets: Prior ambiguity work found 13% of examples ambiguous due to temporal deixis, but this underestimates temporally dependent questions in SituatedQA.Context dependence can coexist with inherent ambiguity, and humans may not treat context-dependent questions as ambiguous when one contextual answer is assumed.
  • Comparison to other QA datasets: Prior temporal QA studies synthetically generate timestamped examples or cloze prompts, whereas SituatedQA manually annotates temporal and geographic questions.The paper characterizes synthetic approaches as limited in scope and diversity.
  • Temporal understanding: Temporal understanding research has typically focused on ordering events or identifying temporal relations within documents.SituatedQA instead studies extra-linguistic temporal context in open-retrieval question answering.
  • Dynamic evaluation: Dynamic evaluation has been studied in language modeling, topic modeling, and entity linking, including temporal drift in pretrained language models.The cited temporal-drift work hints that reusing earlier model snapshots can cause performance decay, as observed here.
  • Dynamic evaluation: The paper proposes dynamically collecting questions to reflect changing facts rather than relying on fixed benchmark snapshots.This extends phased data collection toward continuously identifying and updating temporally dependent facts.
  • Keeping knowledge sources up-to-date: Other work keeps knowledge sources current through knowledge-base revision extraction or entailment decisions based on Wikipedia revisions.These studies address changing knowledge sources rather than the situated QA evaluation introduced here.

8 Conclusion & Future Work

The paper concludes that current open-retrieval QA systems fail to adapt to temporal and geographic shifts. SituatedQA provides tasks and data for modeling changing facts, while future work can broaden sources and context types.

  • Conclusion: Current QA systems fail to adapt to shifts in temporal or geographical context.The conclusion motivates tasks and a dataset for training and evaluating context-sensitive prediction.
  • Conclusion: SituatedQA is intended to support models that update predictions based on new temporal and geographical contexts.The authors describe this as an opportunity for future model development.
  • Future work: Future research may incorporate temporally and geographically dependent source documents such as news articles.The paper also suggests studying additional contexts, including who asks the question and individual preferences.

A Data Collection Details

SituatedQA is constructed by identifying temporally and geographically dependent questions, collecting context-specific answers, and validating the resulting examples. The dataset supports evaluations of adaptation to new contexts and reports annotation agreement and answer consistency with existing data.

  • The dataset includes questions from existing open-retrieval QA sources, including temporal and geographical dependence examples.
  • Annotators collect alternative answers across temporal and geographic contexts and validate them using search and previously selected articles.
  • The collection includes split and source-dataset composition, temporal-volatility distributions, identification agreement, and retrieval evaluations.
  • 70% of temporal and 85% of geographical question-context-answer pairs were judged correct during validation.
  • Around 40% of collected answers matched original Natural Questions answer spans when annotated dates overlapped with the 2018 dataset creation period.The original NQ-Open test agreement rate was 49.2%.

B Data Analysis

The analysis measures naturally occurring temporal and geographical query specifications and examines whether retrieval systems adapt to updated answers and corpora. It finds that retriever performance also declines for queries whose answers have changed since training data collection.

  • 4.1% of NQ-Open questions use query modifications such as “previously” or “last” to specify temporal context.
  • 5.1% of NQ-Open questions specify a point in time through a year expression.
  • 4.4% of NQ-Open questions specify geographical context through a phrase identified from stripped questions.
  • Retrieval performance shows trends similar to end-to-end performance when evaluated on situated question answering.
  • Retriever performance also suffers on queries with updated answers, suggesting implicit temporal anchoring to the training-data period.

D Implementation Details

The implementation uses transformer-based baselines for context identification and question answering, with separate retrieval and reader fine-tuning procedures. Supplementary figures document interfaces for identifying, collecting, searching, viewing, and validating context-answer pairs.

  • Context Dependent Question Identification Baselines: BERT-base and BERT-large classification models are trained for 10 epochs with batch sizes of 8 and 64, respectively.Training uses a learning rate of 5e-5 and 500 warmup steps.
  • Context Dependent Question Answering Baselines: Closed-book baselines are fine-tuned for 10 epochs with batch size 256, AdamW, and learning rate 1e-5.
  • Context Dependent Question Answering Baselines: Retrievers use in-batch negatives plus one hard-negative passage per question, followed by separate reader fine-tuning.Retrievers train for 8 epochs and readers for 16 epochs, both with batch size 16 and learning rate 1e-5.
  • Figures 4–8 show interfaces for identifying dependent questions, collecting and verifying context-answer pairs, searching articles, and viewing selected Wikipedia articles.
Loading 2109.06157v1…