PaperlayerView exact revision on arXiv

Source-linked AI summary

A Dataset for Answering Time-Sensitive Questions

Wenhu Chen, Xinyi Wang, William Yang Wang

arXiv:2108.06314v5cs.CLcs.AI

TL;DR

Time-sensitive questions are underrepresented in existing QA datasets despite the prevalence of evolving facts and the need for temporal reasoning. The paper constructs TimeQA by mining and annotating time-evolving facts linked to Wikipedia, then generating questions. Existing QA models reach 45% on the hard version versus 87% for humans, leaving substantial room for improvement.

  • Problem

    Existing QA datasets contain few questions that require reasoning over modifiable time-sensitive facts.

  • Method

    TimeQA mines time-evolving facts from WikiData, aligns them with Wikipedia text, and uses human verification to construct a temporal QA dataset.

  • Results

    45% accuracy on TimeQA-hard versus 87% human performance shows that existing models lag behind humans on the benchmark.

  • Takeaways & Limitations

    TimeQA provides a benchmark for developing NLP models that better handle temporal shifts and temporal reasoning in text.

  • Takeaways & Limitations

    Implicit temporal commonsense patterns in everyday text remain challenging, motivating algorithms that inject temporal commonsense knowledge into models.

Abstract

from arXiv · show

Time is an important dimension in our physical world. Lots of facts can evolve with respect to time. For example, the U.S. President might change every four years. Therefore, it is important to consider the time dimension and empower the existing QA models to reason over time. However, the existing QA datasets contain rather few time-sensitive questions, hence not suitable for diagnosing or benchmarking the model's temporal reasoning capability. In order to promote research in this direction, we propose to construct a time-sensitive QA dataset. The dataset is constructed by 1) mining time-evolving facts from WikiData and aligning them to their corresponding Wikipedia page, 2) employing crowd workers to verify and calibrate these noisy facts, 3) generating question-answer pairs based on the annotated time-sensitive facts. Our dataset poses challenges in the aspect of both temporal understanding and temporal reasoning. We evaluate different SoTA long-document QA systems like BigBird and FiD on our dataset. The best-performing model FiD can only achieve 46\% accuracy, still far behind the human performance of 87\%. We demonstrate that these models are still lacking the ability to perform consistent temporal reasoning. Therefore, we believe that our dataset could serve as a benchmark to develop NLP models more sensitive to temporal shifts. The dataset and code are released in~\url{https://github.com/wenhuchen/Time-Sensitive-QA}.

1 Introduction

Time-sensitive facts change over time, but existing QA datasets contain few questions requiring models to reason over changing temporal information. TimeQA addresses this gap with annotated time-evolving facts and questions designed to test temporal understanding and reasoning.

  • Motivation: Time-sensitive questions include a modifiable time specifier whose change alters the answer and requires temporal reasoning.Existing datasets often contain few such questions, or use redundant time phrases that do not change the answer.
  • Dataset construction: TimeQA is built by mining time-evolving WikiData facts, aligning them with Wikipedia text, and using human verification before generating question-answer pairs.Figure 2 summarizes fact mining, alignment, and verification as the annotation workflow.
  • Benchmark: TimeQA provides a benchmark focused on temporal understanding and temporal reasoning in long documents.The dataset was motivated by the difficulty of locating scattered temporal transitions expressed in diverse explicit and implicit forms.
  • Challenges: Temporal understanding requires identifying fact time scopes, including implicitly expressed intervals such as the Second World War.Temporal reasoning then relates the query’s time to document times through relations such as within, before, or after.
  • Evaluation: 45% accuracy on the hard version contrasts with 87% human performance, showing substantial difficulty for existing QA models.Across easy and hard versions, model performance drops from 60% to 45%.

2 Dataset and Problem Definition

TimeQA is constructed by mining, aligning, and human-verifying time-evolving facts, then synthesizing questions that require temporal understanding and reasoning. The dataset includes template-generated and human-paraphrased questions across easy and hard reasoning settings.

  • Dataset Construction: The construction pipeline comprises fact annotation followed by question-answer synthesizing.Fact annotation includes mining, aligning, and verifying facts.
  • Fact Annotation: Roughly 150K time-evolving facts are mined from WikiData using temporal qualifiers P580, P582, and P585.Numeric-object triples are discarded because they are unlikely to appear in Wikipedia text.
  • Fact Annotation: Human verification corrects noisy text-fact pairs, including erroneous objects, unanswerable labels, and surface-form mismatches.Workers can correct objects, mark facts unanswerable, or replace an unanswerable label with an object found in the text.
  • Fact Annotation: 5060 golden text–time-evolving-fact pairs remain after filtering 5.5K worker-annotated examples, with facts averaging four segments.The final facts span over 60 relations; 12% of segments are unanswerable and 7% have multiple objects.
  • Question-Answer Synthesis: Questions are generated from relation-specific templates using reasoning types such as in, between, before, and after.Easy questions use time specifiers matching segment boundaries, whereas hard questions use time points within spans and require more implicit temporal reasoning.
  • Dataset Statistics: TimeQA contains 20K questions for both easy and hard versions, while a human-paraphrased subset adds 1171 training and 989 test questions.The paraphrased subset covers 320 training facts and 257 test facts with balanced relations.

3 Models

The paper evaluates BigBird as an extractive model and FiD as a generative model for long-document temporal question answering. BigBird uses sparse local-global attention, while FiD encodes document paragraphs separately and decodes answers autoregressively.

  • BigBird Extractive Model: BigBird predicts answer spans by extracting start and end positions from a concatenated question-document sequence.Its generalized attention addresses sequences that can exceed 4K tokens.
  • BigBird Extractive Model: BigBird combines local sliding-window attention with random attention to capture local and global information.This produces a sparse attention matrix and reduces the quadratic computation cost toward linear.
  • BigBird Extractive Model: BigBird selects the start and end positions by maximizing the product of their predicted probabilities, then returns the intervening sequence span.The predicted answer is Xi:j.
  • FiD Generative Model: FiD splits the document into L short paragraphs, encodes each paragraph with the question, and concatenates their representations.The decoder generates answer tokens autoregressively from the combined paragraph representations.
  • FiD Generative Model: FiD reduces computation to O((K + M)^2L) ≈ O((K + M)N) ≪ O(N^2), enabling processing of input sequences up to 4K tokens.The decoder attends over a shorter sequence, making its attention cost negligible.

4 Experiments

Experiments evaluate BigBird and FiD on TimeQA across training settings, question variants, retrieval, document length, relation types, consistency, and error categories. Models improve with TimeQA adaptation but remain substantially weaker than humans, especially on hard temporal reasoning.

  • Main results: Training only on the limited TimeQA set yields performance below 20%.This setting fine-tunes BigBird and FiD without external NQ or TriviaQA data.
  • Main results: 33% easy-mode and 27% hard-mode accuracy follow NQ/TriviaQA pre-fine-tuning, versus over 60% on NQ/TriviaQA.The gap appears despite little distributional shift because the questions are linguistically simple, natural, and Wikipedia-based.
  • Main results: 60% easy-mode and 45% hard-mode performance result from additional TimeQA fine-tuning, while human hard-mode performance reaches 87%.The best model remains far behind humans, leaving substantial room for improvement.
  • Experimental setup: BigBird and FiD are evaluated with EM and F1 on easy and hard modes, using dev/test splits averaged over three runs.Experiments use HuggingFace Transformers, AdamW, three epochs, and model selection on the dev set.
  • Robustness: Models lose 15% from easy to hard mode, whereas humans lose only 2%, indicating weaker robustness in temporal reasoning.The experiments consistently find lower model accuracy under hard mode.
  • Human-paraphrased results: On human-paraphrased questions, NQ+TimeQA models incur only a 4–5% EM drop without paraphrase training, shrinking to 2% after 1K adaptation examples.The narrow gap between human-written and synthesized questions supports TimeQA as a proxy for real-world time-sensitive QA.
  • Open-domain QA: Open-domain retrieval reaches 28.8% HITS@1 on easy and 26.8% on hard, while end-task QA reaches roughly 14% and 11%, respectively.The pipeline uses BM25 retrieval over Wikipedia followed by BigBird answer extraction.
  • Model analysis: FiD remains more stable across document lengths, while BigBird degrades rapidly beyond 5000 tokens.The analysis attributes part of FiD’s advantage to handling long-term temporal dependencies.

5 Related Work

Related work spans general QA, temporal reasoning over knowledge bases and text, and event-centric temporal tasks. TimeQA differs by emphasizing synthesized temporal questions, implicit temporal mentions, numerical reasoning, and long-term dependencies.

  • Question Answering: General QA datasets progressively challenge machine reading comprehension beyond SQuAD’s typically single-sentence questions.Examples include DrQA, TriviaQA, SearchQA, and DROP.
  • Temporal Reasoning over Knowledge Base: Temporal knowledge-base research adds temporal quantifiers to world knowledge through temporal knowledge extraction.This line of work targets time evolution in structured knowledge bases.
  • Temporal Reasoning over Text: SituatedQA uses more realistic NQ-derived queries, whereas TimeQA contains 20K queries and emphasizes implicit temporal mentions in its hard version.SituatedQA contains 4K temporal queries and does not emphasize the same implicit-temporal challenge.
  • Temporal Reasoning over Events: Compared with TORQUE, TimeQA requires numerical reasoning over time and modeling long-term temporal dependencies in text.TORQUE primarily uses shorter passages and does not require numerical temporal reasoning.

6 Conclusion

The paper presents time-sensitive facts as pervasive but understudied and introduces a dataset for testing whether models understand them. Experiments find a substantial gap between state-of-the-art models and humans, motivating temporal-aware modeling.

  • Time-sensitive facts are pervasive in daily text corpora but have received little prior research attention.
  • TimeQA is presented as the first dataset for investigating whether existing models understand time-sensitive facts.
  • Experiments show state-of-the-art models lag behind humans in temporal reasoning.
  • The paper calls for temporal-aware models and identifies improved modeling of temporal information in text representations as a research direction.

A.1 Dataset documentation and intended uses

The dataset documentation states that the authors follow datasheets-for-datasets guidelines. No further documentation details are supplied in the passage.

  • The authors follow datasheets-for-datasets guidelines to document the dataset.

A.1.1 Motivation

TimeQA was created to test models’ ability to perform diverse temporal reasoning over unstructured text and help future NLP models capture the time dimension.

  • TimeQA targets diverse temporal reasoning under unstructured text corpora.Its intended use is to help future NLP models better capture the time dimension.

A.1.2 Composition

TimeQA contains text documents and roughly 20K question-answer pairs, with randomly split data and potential annotation noise; it also relies on external resources and publicly available samples.

  • TimeQA contains only text documents and roughly 20K question-answer pairs.
  • The dataset is sampled from a larger set, but its representativeness was not established.
  • TimeQA does not define explicit relationships between individual instances.
  • Training, development, and testing sets are split randomly within each data category.
  • Potential noise may remain in question or answer annotations.
  • The dataset relies on external resources, including linked materials whose persistence and archival status require consideration.
  • All TimeQA samples are publicly available and contain no confidential data.
  • The dataset is reported as containing no offensive content and no data relating to people.

A.1.3 Uses

TimeQA is proposed for question answering and is released with code and data through GitHub, while analyses examine its relations and model performance on answerable and unanswerable questions.

  • TimeQA is proposed for QA and may support other tasks such as relation extraction.
  • The dataset and code are released through GitHub without a DOI, under the BSD 3-Clause license.
  • The UCSB NLP group will maintain the dataset on GitHub.
  • Evaluation Metrics: The F1 score includes a formulation intended to cover the unanswerable case.
  • Dataset Construction: The annotation interface is demonstrated in Figure 8.
  • Dataset Composition: Annotated time-evolving facts cover more than 70 relations with a long-tail distribution, dominated by play for, position held, employer of, and educated at.
  • Model Evaluation: FiD exceeds BigBird on unanswerable-question accuracy by 14% in easy mode and 18% in hard mode, while answerable-question gaps are smaller.
Loading 2108.06314v5…