Source-linked AI summary
"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding
Ben Zhou, Daniel Khashabi, Qiang Ning, Dan Roth
TL;DR
Temporal commonsense is important for understanding events but has received limited focused study. The paper defines five temporal properties, builds the crowdsourced MCTACO dataset, and evaluates existing NLP systems on it. Although pretrained methods improve over naive baselines, they remain about 20% behind human performance, motivating further research.
Problem
Temporal commonsense about event duration, frequency, ordering, timing, and persistence is important for language understanding but has received limited attention.
Method
The paper defines five temporal commonsense categories, constructs MCTACO through crowdsourcing, and evaluates ESIM, BERT, and related systems on the dataset.
Results
Existing techniques improve over random-guess baselines but remain far behind human performance on temporal commonsense understanding.
Takeaways & Limitations
MCTACO and the reported model gap provide a testbed and motivation for further research on temporal commonsense understanding.
Takeaways & Limitations
BERT remains brittle to variability in temporal mentions even after unit normalization, and its performance is still poor compared with humans.
Abstract
from arXiv · showhide
Understanding time is crucial for understanding events expressed in natural language. Because people rarely say the obvious, it is often necessary to have commonsense knowledge about various temporal aspects of events, such as duration, frequency, and temporal order. However, this important problem has so far received limited attention. This paper systematically studies this temporal commonsense problem. Specifically, we define five classes of temporal commonsense, and use crowdsourcing to develop a new dataset, MCTACO, that serves as a test set for this task. We find that the best current methods used on MCTACO are still far behind human performance, by about 20%, and discuss several directions for improvement. We hope that the new dataset and our study here can foster more future research on this topic.
1 Introduction
The paper studies temporal commonsense, a limited-attention aspect of natural-language understanding involving properties such as duration, ordering, typical time, frequency, and stationarity. It introduces MCTACO and evaluates existing NLP methods, finding that they remain substantially below human performance.
- Temporal commonsense helps people infer that vacations typically last longer and occur less often than walks, but computers still struggle with such reasoning.
- The paper defines five properties: duration, temporal ordering, typical time, frequency, and stationarity.These cover event length, order, timing, occurrence rate, and whether a state persists indefinitely.
- MCTACO is a crowdsourced dataset for evaluating temporal commonsense through contexts, questions, and potentially multiple plausible candidate answers.The task is framed as binary plausibility classification rather than identification of an absolute truth.
- Existing ESIM, BERT, and related techniques improve over random guessing but remain far behind human performance on MCTACO.The gap indicates that current methods have limited ability to capture temporal semantics.
2 Related Work
Prior commonsense research largely addresses physical properties, event participants, or likely subsequent events, while temporal commonsense has received little focused study. Related temporal-language work handles expressions, relations, and timelines, but does not directly provide a unified temporal-commonsense evaluation.
- Earlier commonsense studies primarily examined physical properties, event participants’ intent and reactions, or likely subsequent events.
- No earlier work had focused specifically on temporal commonsense, despite its importance for event understanding and story timeline construction.Resolving implicit event durations is identified as necessary for constructing a story timeline.
- Temporal-language research commonly extracts and normalizes temporal expressions, extracts temporal relations, or constructs timelines rather than directly evaluating temporal commonsense.Some duration-related work is implicitly connected to temporal commonsense.
- The paper formulates temporal commonsense as machine reading comprehension based on selecting plausible responses to natural-language queries.
3 Construction of MCTACO
MCTACO is built from sentence-question-answer tuples using staged crowdsourcing, automated candidate expansion, and strict annotation checks. The process generates diverse temporal candidates while filtering questions and labels for validity and agreement.
- MCTACO contains 13k sentence-question-candidate-answer tuples drawn from nine MultiRC domains.Questions and candidate answers were collected through Amazon Mechanical Turk from native speakers using qualification tryouts.
- Question generation requires each question to target one of five temporal phenomena and not be answerable from a phrase in the source sentence.Crowdsourcers also provide an initial correct answer.
- Question verification retains only grammatically and logically correct questions whose two reviewers unanimously agree with the initial decision.Valid questions then receive correct and incorrect seed answers.
- BERT supplies replacement terms for masked candidate tokens, retaining its three highest-confidence predictions.
- Event candidates are expanded with similar phrases retrieved from a 60k-event PropBank pool to improve diversity and validity.
- The expansion process targets 20 candidates per question and includes both correct and incorrect answers.
- Four crowdsourcers label each tuple as likely, unlikely, or invalid, and tuples are retained only when all four agree on likely or unlikely.
4 Experiments
The experiments evaluate crowdsourced MCTACO with human and neural baselines using EM and F1, finding substantial gains over naive baselines but a persistent human–system gap. Analyses examine metric behavior, temporal-category variation, and BERT’s difficulty with numerical temporal expressions.
- Experimental setting: The development split is 30% and the test split is 70%, because systems must bring substantial world knowledge beyond task-specific training data.The authors report that development-data gains become marginal after a certain number of training instances.
- Experimental setting: MCTACO evaluation compares native baselines, ESIM, BERT variants, and human performance using question-level EM and F1.EM requires labeling all candidate answers correctly, whereas F1 measures prediction overlap with the ground truth.
- Results: ELMo and BERT improve naive baselines by a large margin, but BERT remains far behind human performance on MCTACO.Across three runs, the maximum standard error is 0.8; BERT + unit normalization scores F1 = 72 and EM = 45 on the 100-question human subset.
- Analysis: BERT often associates temporal surface forms without distinguishing numerical magnitudes, such as treating 1 hour as likely but 9 hours as unlikely.The authors interpret this behavior as selecting individual options rather than inferring a range of true answers.
- Analysis: EM is consistently lower than F1, and systems other than humans show an approximately 30% drop from F1 to EM.Humans show an 11.3% drop, while BERT + unit normalization is not uniform across temporal categories and remains poor compared with humans.
5 Conclusion
The paper defines temporal commonsense categories, develops MCTACO through crowdsourcing, and evaluates current systems on temporal commonsense understanding. State-of-the-art language models remain far behind human performance, motivating further research.
- The paper defines five categories of questions requiring temporal commonsense and develops MCTACO through a novel crowdsourcing scheme.
- MCTACO is used to probe the temporal commonsense capabilities of current systems.
- Systems equipped with ELMo and BERT remain far behind human performance on temporal commonsense understanding.
- The analysis describes both the capabilities and limitations of current models and motivates future research.
A.1 Perturbing Candidate Answers
The annotation process creates candidate-answer perturbations using collections of common temporal expressions and their variations.
- Common temporal expressions are collected to detect whether candidate answers contain temporal expressions.
- Detected temporal expressions are perturbed using mappings between expressions and their variations.
- Table 3 groups each temporal mention with its variations, such as “first” and “last.”
A.2 Performance as a function of training size
The paper argues that temporal commonsense requires external world knowledge and examines how supervised training size affects performance. Performance plateaus after about 2.5k question-answer pairs when pre-training supplies world knowledge.
- External world knowledge is needed because a dataset is unlikely to cover all possible events and their attributes.
- Traditional supervised learning without pre-training or external training is unlikely to succeed on this task.
- Performance plateaus after about 2.5k question-answer pairs, approximately 20% of the datasets.
- The result supports relying on a relatively small amount of supervision to tune a task when world knowledge is modeled through pre-training.
A.3 Annotation Interfaces
The figures document the annotation workflow through three successive steps. Figure 3 separately shows supervised BERT performance as training-data size varies, with no-data performance at random guessing.
- Figure 3 plots supervised BERT performance as a function of the size of observed training data.
- With no training data, the performance measures amount to random guessing.
- Figure 4 is labeled Step 1 in the annotation workflow.
- Figure 5 is labeled Step 2 in the annotation workflow.
- Figure 6 is labeled Step 3 in the annotation workflow.