PaperlayerView exact revision on arXiv

Source-linked AI summary

TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions

Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, Dan Roth

arXiv:2005.00242v2cs.CL

TL;DR

Temporal relations are often implicit, yet reading-comprehension benchmarks study them very little. TORQUE pairs news snippets with human-generated temporal questions; RoBERTa-large achieves 51% exact-match, about 30% behind humans.

  • Problem

    Reading-comprehension benchmarks study temporal phenomena very little, despite the need to answer questions about relationships between events that may be implicit in text.

    Source: §1 · p. 1

  • Method

    TORQUE pairs 3.2k TempEval3 snippets with crowd-labeled events and human-written, answered temporal questions, including contrastive changes to temporal keywords.

    Source: §1 · p. 2

  • Results

    51% exact-match was achieved by fine-tuned RoBERTa-large on TORQUE, about 30% behind human performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The performance gap supports further investigation into improving machine reading comprehension for temporal relationships.

    Source: §8 · p. 9

    The paper focuses on constructing a QA dataset; transfer to downstream tasks outside QA may require special modeling and remains out of scope.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

A critical part of reading is being able to understand the temporal relationships between events described in a passage of text, even when those relationships are not explicitly stated. However, current machine reading comprehension benchmarks have practically no questions that test temporal phenomena, so systems trained on these benchmarks have no capacity to answer questions such as "what happened before/after [some event]?" We introduce TORQUE, a new English reading comprehension benchmark built on 3.2k news snippets with 21k human-generated questions querying temporal relationships. Results show that RoBERTa-large achieves an exact-match score of 51% on the test set of TORQUE, about 30% behind human performance.

1 Introduction

Temporal relations are rarely studied in reading comprehension, although they can be implicit and require reasoning across events. TORQUE targets this gap, and RoBERTa-large remains about 30% behind human performance.

1 Introduction

  • Temporal phenomena are rarely studied in reading comprehension, and existing systems are brittle on TORQUE questions.Temporal relations can be implicit, as when text indicates an event is future or past without stating an explicit time.Source: §1 · p. 1
  • Answering temporal questions requires identifying events and their times, then reasoning across the passage rather than relying only on local predicate-argument structures.The passage-level examples include multiple valid answers such as SNOW and RAINFALL.Source: §1 · p. 2
  • A temporal question may have multiple answers or none when its queried time falls outside the passage's scope.For example, a question about what happened before the snow started may have no answer.Source: §1 · p. 2
  • 51% exact-match was achieved by fine-tuned RoBERTa-large on TORQUE, about 30% behind human performance.The benchmark's test data was drawn from a 20% validation split.Source: §1 · p. 2
  • TORQUE introduces the first reading-comprehension benchmark for temporal relations, with 3.2k snippets and 25k events paired with 21k human-generated, answered questions.The snippets were randomly selected from TempEval3; workers also wrote contrasting questions to reduce data-collection artifacts.Source: §1 · p. 2

2 Definitions

The paper treats events broadly, including verbs and nouns and events regardless of modality. It replaces fixed interval labels with natural-language descriptions of temporal relationships.

2.1 Events

  • Events are defined as verbs or nouns, with the verb labeled in copular constructions; events of any modality are annotated.This choice gives crowd workers a consistent labeling rule, while dependency parsing or semantic role labeling can recover links to associated adjectives.Source: §2.1 · p. 3

2.2 Temporal Relations

  • Prior work assigns events intervals and compares them with 13 relation labels, but fuzzy, nonfactual, or repetitive events can exceed that scheme.The paper therefore moves from a fixed relation set to natural-language annotations of relationships.Source: §2.2 · p. 3

3 Natural Language Annotation of Temporal Relations

TORQUE frames temporal relations as natural-language questions rather than a fixed set of relation labels. This format accommodates fuzzy timing, differing event modes, and contrast questions designed to penalize shortcuts.

3 Natural Language Annotation of Temporal Relations

  • Temporal relation questions ask for all passage events A satisfying relation r to event B, with passage truth and writing time as reference assumptions.The paper denotes the query as (?, r, B).Source: §3 · p. 4

3.1 Advantages

  • Fuzzy timing, such as events occurring at about the same time with unknown order, can be captured naturally in question-answer pairs.The text gives DISRUPTION and FLOODING as an example of roughly simultaneous events whose order is uncertain.Source: §3.1 · p. 4
  • Natural-language questions can distinguish event modes that a single relation label would collapse into the same “before” relation.Repetitive events may span multiple intervals and carry different semantics.Source: §3.1 · p. 4
  • Questioning naturally avoids unrelated event pairs, bypassing the need to annotate event properties or build theories for deciding which pairs have temporal relations.The examples contrast unnatural questions about unrelated events with natural questions about expected or intended events.Source: §3.1 · pp. 4–5
  • Transfer to downstream tasks outside question answering may be difficult and may require special modeling; successful transfer is outside this paper’s scope.Source: §3.1 · p. 5

3.2 Penalize Shortcuts by Contrast Sets

  • Contrast questions change temporal keywords or event boundaries while requiring different answers, penalizing shortcuts based on question wording.Examples change before/after or ask about an event’s start or end.Source: §3.2 · p. 5

4 Data Collection

TORQUE was collected from two-sentence passages sampled from TempEval3 articles, using workers to label events and create temporal questions and answers. Qualification, piloting, and validation controlled annotation quality.

4 Data Collection

  • Workers annotated randomly sampled two-sentence passages from a pool of 26k passages drawn from 2.8k TempEval3 articles.The authors selected two contiguous sentences as sufficient to capture the vast majority of non-trivial temporal relations.Source: §4 · p. 5
  • For each passage, workers labeled events, asked temporal-relation questions, identified answers, and modified questions to create additional question-answer pairs.Source: §4 · p. 5

4.1 Quality Control

  • Workers passed qualification and pilot review before large-scale annotation; 63 reached level 2, and 60 worked on the full task.Qualification tested event labeling, question writing, and question answering; pilot submissions were manually checked with feedback.Source: §4.1 · p. 6
  • For 20% of articles, workers validated events and answers with added noise; event lists used majority vote, while all workers’ final answers were treated as correct.Questions were not additionally validated after a manual inspection found no bad questions in a random sample of 100.Source: §4.1 · p. 6

4.2 Cost

  • Each job offered $6 base pay plus $0.5 per extra question; workers used about 2 of 3 passages on average, and the dataset cost $15k ($0.70/question).Validation of events and answers cost $0.1 each.Source: §4.2 · p. 6

5 TORQUE Statistics

TORQUE contains 3.2k passage annotations, 24.9k events, and 21.2k user-provided questions. Its event and answer annotations were assessed through worker validation and noise checks.

5 TORQUE Statistics

  • TORQUE contains 3.2k passage annotations, 24.9k events, and 21.2k user-provided questions, about half modified from existing questions.Passages average about 50 tokens and 7.9 events; sampling with replacement yielded 2.9k unique passages.Source: §5 · p. 6, §5.1 · p. 6
  • In a random sample of 200 test questions, 94 queried relations not directly representable by prior single-interval labels.The table defines “standard” relations as those directly captured by the previous single-interval label set; some questions have multiple types.Source: §5 · p. 6, Table 2 · p. 7

5.1 Quality

  • Event annotations had 94.2% WAWA F1; validators retained 99.4% of candidates labeled by both the initial annotator and model, but only 0.8% labeled by neither.The validation setup added noise to the initial event annotations.Source: §5.1 · pp. 6–7
  • Answer annotations had 84.7% WAWA F1; validators marked 12.3% of randomly added candidates correct, close to the 11.6% found in manual inspection.The manual inspection covered 100 questions.Source: §5.1 · p. 7

6 Experiment

RoBERTa-large remains about 30% behind human performance in exact match on TORQUE, despite the benchmark’s current training-set size appearing not to be the main bottleneck. The experiments evaluate token-level answer classification with BERT and RoBERTa models on article-separated train, development, and test splits.

6 Experiment

  • The model classifies each passage token as an answer or not using a one-layer perceptron over BERT or RoBERTa representations, with train, development, and test articles kept separate.BERT and RoBERTa base and large variants were fine-tuned; the human estimate came from one author answering 100 test questions and comparison with crowd annotations.Source: §6 · p. 7
  • With 10% of the training data, RoBERTa predicts “not an answer” for every token; at 50%, its performance is slightly lower than but comparable to using all the data.The authors conclude that the learning curve is already flat and the dataset’s current size may not be the bottleneck for low performance.Source: §6 · pp. 7–8
  • About 30% behind human performance, RoBERTa-large achieves the best system result in exact match on TORQUE’s test set.The reported human and system results are in Table 4.Source: §6 · p. 7, Table 4 · p. 8

7 Related Work

Prior temporal-relation work has pursued algorithmic models and formalized annotation schemes, while this paper frames temporal understanding as reading comprehension. Related QA approaches differ in task focus and in how their data are collected.

7 Related Work

  • Temporal-relation modeling has used structured inference, structured learning, and neural networks.Examples include Do et al. (2012), Chambers et al. (2014), Ning et al. (2018a), Leeuwenberg and Moens (2017), and Dligach et al. (2017).Source: §7 · p. 8
  • Earlier formalism-based work defined event modes, label sets, time axes, or decision rules, with annotation agreement varying by the breadth of phenomena covered.Bethard et al. (2007) and Ning et al. (2018b) focused on limited phenomena and achieved high agreement, whereas broader-coverage efforts reported low agreement even among NLP researchers.Source: §7 · p. 8
  • QA-SRL and QAMR use QA pairs to represent predicate-argument structures, while TORQUE uses temporal relations between events as its focus.The paper also distinguishes TORQUE from zero-shot relation extraction, which centers on entities and uses a different question design and crowdsourcing pipeline.Source: §7 · p. 8
  • QA-TempEval evaluates systems that generate TimeML annotations rather than posing a reading-comprehension task, and depends on an existing TimeML corpus.The paper contrasts this with TORQUE’s task definition and scalable crowdsourcing pipeline, which can be adopted to collect more data.Source: §7 · pp. 8–9

8 Conclusion

TORQUE is an English reading-comprehension dataset for temporal ordering, built from news snippets with hard-coded and human-generated questions. RoBERTa-large remains far behind human performance, motivating further investigation of temporal-relation reading comprehension.

8 Conclusion

  • RoBERTa-large falls behind human performance by a large margin on TORQUE, prompting further investigation into temporal-relation reading comprehension.The conclusion characterizes this as a need for more investigation into improving MRC on temporal relationships.Source: §8 · p. 9
  • TORQUE contains 3.2k news snippets, 9.5k hard-coded questions, and 21.2k human-generated questions about temporal ordering.The hard-coded questions ask which events have finished, are ongoing, or remain in the future; human-generated questions query more complex phenomena.Source: §8 · p. 9
  • The paper argues that an MRC setting represents temporal phenomena more conveniently than conventional formalisms.Source: §8 · p. 9

A Event Distribution

The appendix characterizes TORQUE’s events, question forms, answer counts, and annotator workload. Its distributions show varied event triggers and question outcomes, with annotations spread across workers.

A Event Distribution

  • TORQUE has 24.9k events over 3.2k passages; the distribution roughly follows a Gaussian, with a mean of around 7–8 events per passage, and reporting and copular verbs are most common.Source: A Event Distribution · p. 12

B Question Prefix Distribution

  • Almost all crowd questions begin with “what”; past questions more often ask what happened before or after, while future questions favor what will happen after.The question-prefix visual includes original questions and their modifications.Source: B Question Prefix Distribution · pp. 12–13, Figure 14 · p. 13

C Answer Distribution

  • Modified questions have a larger share of zero-answer cases than original questions, especially among warm-up questions in news articles where most events are past.For user-provided questions, the increase in zero-answer cases is less drastic because modifications only had to produce different answers.Source: Figure 15 · p. 13, C Answer Distribution · p. 13

D Workload Distribution Among Workers

E Worker Agreement With Aggregate

WAWA measures inter-annotator agreement by comparing workers’ answers with an aggregate answer set. Tables 5 and 6 report agreement for event and question-answer annotations, with a modified comparison for the initial annotator.

E Worker Agreement With Aggregate

F Reproducibility

The paper reports the software, hardware, and model configuration used for reproducibility.

F Reproducibility

  • The experiments used PyTorch 1.3.1 and Huggingface transformers for pretrained language models.Source: F Reproducibility · p. 15
  • A GeForce RTX 2080 GPU was used for finetuning, and a full TORQUE training epoch with RoBERTa-large averaged 38 minutes.The reported CUDA version was 10.2.Source: F Reproducibility · p. 15
  • The best-performing model was RoBERTa-large with a final MLP layer, comprising 355M + 1024 * 64 + 64 * 2 parameters.Source: F Reproducibility · p. 15

Related papers

Loading 2005.00242v2…