PaperlayerView exact revision on arXiv

Source-linked AI summary

Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, Bryan Perozzi

arXiv:2406.09170v1cs.CL

TL;DR

Temporal reasoning benchmarks can conflate reasoning with prior knowledge, shortcuts, or flawed anonymization. ToT introduces controlled semantic and arithmetic tasks, finding performance differences across temporal structures and exposing distinct model weaknesses while releasing the datasets and evaluation framework.

  • Problem

    Existing temporal benchmarks incompletely cover temporal reasoning and may permit prior-knowledge shortcuts or introduce inconsistencies through anonymization.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    ToT uses a synthetic semantic dataset and a crowdsourced arithmetic dataset to evaluate temporal logic and temporal calculations independently.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    LLM performance varies substantially with temporal structure, with GPT-4 ranging from 20.33% on complete graphs to 90.83% on AWE graphs.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    ToT provides a controlled benchmark for analyzing strengths and weaknesses in LLM temporal reasoning and supports further research through open-sourced resources.

    Source: §5 · pp. 9–10

    ToT-Arithmetic includes a recurring leap-year calculation failure when dates span multiple years.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.

1 Introduction

Temporal reasoning is important for intelligent systems, but existing benchmarks often use knowledge-graph facts that can reward prior knowledge or shortcuts instead of genuine temporal reasoning. ToT addresses these limitations with controlled tasks covering temporal semantics and arithmetic.

  • Existing temporal benchmarks often rely on knowledge-graph-style facts about well-known entities, limiting coverage of temporal relationships.Source: §1 · p. 1
  • Such benchmarks can reflect models’ prior knowledge or shortcuts rather than explicit temporal reasoning, while omitting broader temporal structures and time arithmetic.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • ToT separates temporal reasoning into semantic logic and temporal arithmetic, enabling controlled evaluation along both dimensions.Source: §1 · p. 2

2 Related work

Prior temporal benchmarks emphasize knowledge graphs and can suffer from costly collection, leakage, spurious correlations, and hallucinated facts. ToT instead uses controlled synthetic generation to test reasoning from presented facts.

  • Temporal reasoning research includes model training strategies and benchmarks, with many benchmarks centered on knowledge graphs.Source: §2 · p. 2
  • Crowd-sourced and anonymized benchmarks can be resource-intensive or introduce leakage, spurious entity correlations, factual inconsistencies, and hallucinated facts.Source: §2 · p. 2
  • ToT extends synthetic reasoning probes to temporal reasoning and controls graph structures so models must reason over presented facts rather than latent knowledge.Source: §2 · p. 3

3 ToT: A benchmark for evaluating LLMs on temporal reasoning

ToT evaluates temporal semantics and temporal arithmetic as distinct skills through complementary datasets. ToT-Semantic uses controllable graph-based questions, while ToT-Arithmetic is built from crowdsourced and functionalized time-calculation problems with repeated quality checks.

3 ToT: A benchmark for evaluating LLMs on temporal reasoning

  • ToT decouples temporal semantics and logic from temporal arithmetic to benchmark strengths and weaknesses along independent axes.Source: §3 · p. 3

3.1 ToT-Semantic: A Synthetic Dataset

  • ToT-Semantic generates diverse graph structures, including ER, scale-free, BA, SBM, star, complete, and anonymized Wikidata graphs, with 5–30 nodes.Source: §3.1 · p. 3
  • ToT-Semantic assigns relation types and valid time intervals to graph edges, then creates eight temporal question types spanning single- and multi-fact reasoning.Source: §3.1 · pp. 3–4
  • The semantic benchmark contains 2,800 questions from seven graph generators, eight question types, five fact orderings, and ten samples per combination.Source: §3.1 · p. 4

3.2 ToT-Arithmetic: A Temporal Arithmetic Dataset

  • ToT-Arithmetic targets practical time calculations using questions expanded by 15 annotators, filtered for corner cases and extensive knowledge requirements, and grouped into seven categories.Source: §3.2 · p. 4, §3.2.1 · p. 5

3.2.1 Task Creation

  • The arithmetic questions were functionalized for varied sampling, producing 1,850 examples across categories including AddSubtract, Compare, Duration, Schedule, Timezone, Trick, and MultiOp.Source: §3.2.1 · p. 5

3.3 Quality Check

  • Both tasks underwent repeated checks of label correctness, question clarity, and answer-format instructions until no further issues were found.Source: §3.3 · p. 5

4 Experiments and Results

Experiments evaluate how graph structure, graph size, question type, fact order, and temporal reasoning axis affect LLM performance. Results show strong sensitivity to structure and ordering, with distinct strengths and failure modes across semantic and arithmetic tasks.

4.1 Investigating the impact of temporal structure on LLM temporal reasoning

  • GPT-4 accuracy varied from 20.33% on complete graphs to 90.83% on AWE graphs, showing that temporal graph structure strongly affects performance.Table 3 reports this graph-structure comparison; the text also describes accuracy as more than doubling between complete and AWE graphs.Source: §4.1 · pp. 6–7, Table 3 · p. 6

4.1.1 Influence of graph size on LLM performance

  • Increasing either the number of edges or nodes in ToT-Semantic decreases LLM performance.The graph-size analysis treats edges as facts and nodes as entities.Source: §4.1.1 · p. 7, Figure 4 · p. 7
  • Graph structure remains important beyond graph size: node counts show no consistent effect across structures, whereas edge counts correlate with performance but have exceptions.SFN graphs slightly outperform Star graphs despite having far fewer average edges, indicating that structure and size both matter.Source: §4.1.1 · p. 7, Table 4 · p. 7

4.2 Effects of temporal question type on LLM temporal reasoning

  • LLMs perform better on single-fact retrieval than on multi-fact questions requiring information integration and operations such as counting or sorting.EventAtTimeT and EventAtWhatTime are single-fact tasks; other question types require multiple facts and additional synthesis.Source: §4.2 · pp. 7–8, Table 5 · p. 7
  • Timeline questions are the most difficult, and GPT-4’s higher accuracy than Claude-3-Sonnet coincides with lower precision and recall because it often outputs too few entities.Gemini 1.5 Pro has superior timeline precision and recall, while complete graphs reduce both precision and recall for Claude-3-Sonnet and GPT-4 and primarily recall for Gemini.Source: §4.2 · p. 8, Table 6 · p. 8

4.3 Impact of temporal fact order on LLM performance

  • Fact order substantially affects performance: shuffled facts yield the lowest results, while TargetAndStartTime ordering yields the highest.The reported implication is that organizing facts according to temporal or task-relevant order can improve accuracy.Source: Table 7 · p. 8, §4.3 · p. 9

4.4 Temporal semantics vs temporal arithmetic

  • ToT-Arithmetic models perform best on Timezone questions and worst on Duration questions, while Scheduling and Trick questions also remain challenging.AddSubtract performance is relatively strong; Duration errors often approximate the answer but miss by one day, with approximately 21% of GPT-4 and 25% of Gemini 1.5 Pro errors within one day.Source: §4.4 · p. 9, Table 8 · p. 9
  • GPT-4 performs better on arithmetic, whereas Gemini 1.5 Pro performs better in the reasoning category; arithmetic failures include direction and leap-year errors.The reported failure patterns include incorrect month direction and inaccurate day counts across leap years.Source: §4.4 · p. 9

5 Conclusion

The paper introduces Test of Time (ToT) as a comprehensive, controlled benchmark for evaluating temporal reasoning in LLMs. Its two-pronged design yields insights into current models’ strengths and weaknesses, while open-sourced resources support further research.

  • ToT is a novel benchmark designed to assess LLMs’ temporal reasoning abilities more comprehensively and controllably than existing work.Source: §5 · pp. 9–10
  • ToT evaluates two complementary aspects of temporal reasoning through semantic and arithmetic tasks.Source: §5 · p. 9
  • The experiments provide insights into current LLMs’ strengths and weaknesses in temporal reasoning.Source: §5 · p. 10
  • The datasets and evaluation framework are open-sourced to stimulate further research and development.Source: §5 · p. 10

A Detailed comparison of ToT with existing benchmarks.

ToT is compared with existing temporal reasoning benchmarks across six dimensions covering question types, data origin, training-data leakage, and implicit reasoning. The analysis reports that ToT uniquely combines all six properties while mitigating training-data leakage.

B Description of graph generators.

The benchmark generates temporal knowledge graphs using six random graph generators and anonymized Wikidata ego-graphs, with temporal intervals attached to graph edges. Its output instructions were usually followed, supporting evaluation focused on temporal reasoning.

B Description of graph generators.

C Quality of instructions

  • Instruction-missing rates were 0.05% for Gemini 1.5 Pro, 0.65% for GPT-4, and 4.76% for Claude-3-Sonnet, supporting a benchmark focused on temporal reasoning.Source: C Quality of instructions · p. 16

D Large-Scale ToT-Semantic Experiments

The expanded ToT-Semantic dataset increases scale and confirms that graph structure, question type, and fact order affect performance. The accompanying annotation instructions target broad coverage of time-arithmetic categories and varied examples.

D Large-Scale ToT-Semantic Experiments

E Instructions to participants

  • The annotation process sought broad coverage of time-arithmetic categories, subcategories, and diverse real-life examples, prioritizing new categories and underrepresented areas.Source: E Instructions to participants · pp. 17–18

F Limitation and future work

The benchmark’s scope is limited by single-sentence temporal anchoring and an intentional focus on explicit temporal facts. The authors also identify risks involving misuse, unequal access, and privacy.

F Limitation and future work

  • The benchmark places both endpoints of each fact within one sentence, whereas real-world temporal information may span multiple sentences or documents.The authors state that this setup could be adapted to multi-sentence scenarios in future work.Source: F Limitation and future work · pp. 18–19
  • The benchmark intentionally excludes static facts without time anchors, limiting evaluation to explicit temporal reasoning rather than combined temporal and general factual reasoning.Future work could add static facts for broader evaluation.Source: F Limitation and future work · p. 19

G Potential negative societal impacts

  • More capable temporal reasoning systems could be misused for fabricated timelines, unequal information access, or privacy violations.The authors present these as potential negative societal impacts requiring responsible development.Source: G Potential negative societal impacts · p. 19

H Dataset documentation: datasheet

The datasheet presents ToT as a controlled benchmark created to address limitations in existing temporal-reasoning evaluations. It consists of text-only, labeled, self-contained instances with a test-only evaluation split.

H.1 Motivation

  • ToT was created to provide a more comprehensive and controlled assessment of LLM temporal reasoning abilities and support more rigorous evaluation.The dataset was created by ML researchers at Google and funded by Google.Source: §H.1 · p. 19

H.2 Composition

  • The benchmark contains text-only sentences describing reasoning problems, with 46,480 ToT-Semantic instances, 2,800 ToT-Arithmetic instances, and a 2,800-example Semantic sample.The larger and smaller sets support different experimental scales.Source: §H.2 · pp. 19–20
  • Each instance has a ground-truth label, complete text and label information, and no relationship to other instances.The benchmark provides only a test split because it is intended for evaluation.Source: §H.2 · p. 20
  • The dataset is self-contained and contains no offensive, subpopulation-identifying, directly identifying, sensitive, or confidential information.These properties are stated in the datasheet’s composition documentation.Source: §H.2 · p. 20

H.3 Collection

The dataset was generated primarily through software with author-curated seed questions and reproducible sampling for a smaller subset. It is intended for temporal-reasoning evaluation and is publicly distributed under CC BY 4.0.

H.3 Collection

  • Each temporal-reasoning question was generated automatically using code, combining author-developed software with human curation of seed questions.The authors describe this combination as providing efficiency and human oversight.Source: §H.3 · pp. 20–21
  • The smaller experimental subset was created by random sampling with a fixed seed to support reproducibility and consistent comparisons.The primary dataset was not derived from another dataset.Source: §H.3 · p. 21
  • The dataset was collected by the paper’s authors, who were full-time Google employees, over four months from March through June.The authors received their regular salaries and no additional task-specific payment.Source: §H.3 · p. 21
  • The documentation reports no consent-revocation mechanism, impact analysis, or ethical review process.These disclosures concern the documented collection process.Source: §H.3 · p. 21

H.4 Uses

  • The dataset is used for temporal-reasoning evaluation, can support related temporal-reasoning tasks, and is distributed publicly through Hugging Face with generation code available on GitHub.The datasets were released under a Creative Commons Attribution 4.0 International license.Source: §H.4 · p. 21, §H.5 · pp. 21–22

J Reproducibility

The reproducibility experiments evaluate three prominent model families using specified model variants and a common cloud infrastructure. The setup uses publicly accessible models to support further research.

J Reproducibility

  • The experiments cover Gemini, Claude, and GPT-4 model families.The paper identifies these as three prominent model families used for reproducibility experiments.Source: J Reproducibility · p. 22
  • The evaluated variants are Gemini 1.5 Pro, Claude’s Sonnet model, and GPT-4 Turbo.Source: J Reproducibility · p. 22
  • The experiments ran on Google Cloud Platform, and the evaluated model variants were publicly accessible through various platforms.The authors cite Google Cloud Platform’s machine-learning tools as the infrastructure rationale.Source: J Reproducibility · p. 22

Related papers

Loading 2406.09170v1…