Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

• About 48% of Wikidata qualifiers are time-related, making temporal reasoning important for time-dependent question answering.Source: §1 · p. 1
• Prior TSQA resources lacked systematic analysis and were limited in time span or temporal relation coverage, motivating a broader benchmark.Source: §1 · pp. 1–2
• Humans’ temporal understanding comprises time-time, time-event, and event-event relations, requiring logic, temporal grounding, retrieval, or relational cues.Source: §1 · pp. 1–2
• LLMs are biased toward contemporary years and perform poorly on low-resource years, with time-expression co-occurrences contributing to this variation.Source: §1 · p. 2
• TEMPREASON spans longer time ranges and all three temporal understanding types, evaluating models in closed-book, open-book, and reasoning QA settings.Source: §1 · p. 2
• Temporal span extraction and time-sensitive reinforcement learning encourage temporally correct answers, and the resulting model consistently outperforms strong baselines.Source: §1 · p. 2

• The study divides 1900 to 2040 into seven 20-year time periods and randomly generates 400 questions for each period to evaluate temporal reasoning in three language models.Source: §2 · p. 3
• Answers are evaluated with exact match, mean absolute error, and trend accuracy, which measures whether predictions fall before or after the reference year.Source: §2 · p. 3
• ChatGPT achieves 99.6 overall EM on year prediction, whereas T5-L-NQ and FLAN-T5-L are biased toward contemporary time ranges.Source: §2 · p. 3, Table 2 · p. 3
• T5-L-NQ and FLAN-T5-L generally have EM scores around 20–30 outside contemporary ranges, revealing weak numeric estimation despite understanding the question format.Source: §2 · p. 3
• FLAN-T5-L has lower EM than T5-L-NQ but lower MAE across most ranges, indicating improved numeric estimation from instruction tuning.Source: §2 · p. 3
• All models exceed 90% on before/after trends before 2020, but T5-L-NQ and FLAN-T5-L generalize poorly to 2020–2040.Source: §2 · p. 3

3.1 TEMPREASON Dataset

• TEMPREASON broadens temporal QA beyond prior datasets’ narrow time spans and frequent-answer bias by targeting systematic temporal reasoning.TEMPLAMA covers 2010–2020, while 70.69% of its questions have the most frequent answer for a subject.Source: §3.1 · p. 4
• The dataset covers L1 time-time, L2 time-event, and L3 event-event reasoning, with L1 questions extending from year to month prediction.L2 and L3 questions are generated from time-dependent Wikidata facts and temporal relations.Source: §3.1 · p. 4, Figure 2 · p. 5
• TEMPREASON retains subject-relation groups with at least three chronological facts, samples L2 query times between fact start and end times, and generates L3 questions from one-hop before/after pairs.It uses ten time-sensitive relations and keeps at most 2,000 subjects for each relation type.Source: §3.1 · p. 4

3.2 Problem Settings

• TSQA evaluates answers with token-level F1 and exact match across answer retrieval, time grounding, and temporal reasoning challenges.TEMPREASON questions contain approximately 5.3–5.5 possible answers per question.Source: §3.2 · pp. 4–5
• The benchmark compares closed-book, open-book, and reasoning QA, respectively requiring memorization, textual context use, or deduction from structured time ranges.Reasoning QA randomly reorders relevant facts to reduce shortcut learning and resembles human temporal reasoning.Source: §3.2 · p. 5

Temporal Training Framework

• The framework sequentially applies temporal span extraction pre-training, TEMPREASON supervised fine-tuning, and time-sensitive reinforcement learning.Temporal span extraction masks temporal and entity spans in 100K Wikipedia articles before task-specific fine-tuning.Source: §4 · pp. 5–6

Time-Sensitive Reinforcement Learning

• TSRL forms negative answers from other time periods in the same subject-relation group and trains the model to prefer the temporally correct answer.These alternatives provide temporally wrong candidates for the reward function.Source: §4 · p. 6, Figure 3 · p. 6
• The reward uses exact-match scores for the ground truth and negative candidates, assigning negative score when the prediction is temporally wrong.The final reward gives positive values to ground-truth matches and negative values to matches with the negative-answer set.Source: §4 · p. 6
• The final TempT5 model is optimized with Proximal Policy Optimization after reward-based temporal fine-tuning.The reward function explicitly distinguishes correct answers from temporally incorrect alternatives.Source: §4 · p. 6

5.2 Experimental Results

• ChatGPT reaches 99.6 EM on year prediction but only 30.5 EM on month prediction, whereas fine-tuned T5-SFT and TempT5 reach 100 EM/F1 on the L1 task.TempT5 achieves 97.1 F1 on the out-of-domain futuristic test set, below its in-domain performance.Source: §5.2 · p. 7
• TempT5 consistently outperforms the strongest baseline, T5-SFT, in OBQA and ReasonQA.The reported experimental results average scores over three runs, with ΔF1 defined as the difference between TempT5 and T5-SFT.Source: §5.2 · p. 7, Table 4 · p. 7
• ReasonQA performs best, OBQA follows, and CBQA performs worst across L2 and L3 temporal understanding.The results indicate that models reason better when relevant facts and timestamps are provided, whereas answering without context is highly difficult.Source: §5.2 · p. 7
• L3 temporal reasoning is more challenging than L2 for most models, with FLAN-T5-L dropping 18.8 F1 from 66.3 to 47.5, except ChatGPT.P39 position-held questions enable a shortcut: changing “46th” to “45th” can answer some questions.Source: §5.2 · pp. 7–8

5.3 Ablation Study

• Removing TSRL reduces ReasonQA by 1.2 F1, while removing TSE pre-training reduces OBQA by 0.8 F1.TSRL has the larger effect in ReasonQA, whereas TSE pre-training is more important in OBQA.Source: §5.3 · p. 8, Table 7 · p. 8

5.4 Further Analysis

• TempT5 reduces temporal bias from 1940 to 2020, but performance remains significantly worse before 1900 and after 2020 because training data are scarce.The model’s time-period performance is similar across 1940–2020, while low-resource periods remain difficult.Source: §5.4 · p. 8, Table 8 · p. 8
• Intra-year questions underperform inter-year questions, with EM scores of 6.4, 86.9 versus 80.5.These questions require month-level reasoning because multiple answers may occur within one year, and the model can miss an intra-year position change.Source: §5.4 · p. 8, Table 9 · p. 8, Table 10 · p. 8

Temporal Information Extraction

• Temporal information extraction datasets annotate events, times, and temporal relations, but their models generally do not perform global event-to-time grounding.TimeBank, TimeBank-Dense, and TempEval exemplify document-level temporal annotation and extraction.Source: §6 · pp. 8–9

Temporal Reasoning over KGs

• Temporal KGQA ranks entities satisfying temporal queries, assuming that the knowledge-graph entities are known to the system.The paper’s task instead answers temporal questions from natural text input only.Source: §6 · p. 9

Temporal Reasoning for LMs

• Existing time-sensitive QA datasets vary in coverage but commonly omit temporal reasoning levels beyond time-event questions.TEMPLAMA and StreamingQA cover relatively short, contemporary periods, while the Time-sensitive QA dataset spans 1367–2018 but focuses on time-event relations.Source: §6 · p. 9

• LLMs are susceptible to temporal bias, and their reasoning capability varies with the specific time in a question.Source: §7 · p. 9
• TEMPREASON evaluates temporal reasoning across diverse settings, while the proposed training paradigm combines temporal span extraction pre-training with time-sensitive reinforcement learning.Source: §7 · p. 9
• Extensive experiments show that the proposed model consistently outperforms strong baselines.Source: §7 · p. 9

• TEMPREASON uses only textual months and numeric years, lacking adversarial attacks with all-numeric dates and months.Source: §8 · p. 9
• Because its question-answer pairs derive from Wikidata triples, knowledge-base errors may propagate into benchmark answers.The authors state that this has minimal effect in ReasonQA, which asks models to infer from Wikidata factual knowledge.Source: §8 · p. 10

9 Ethics Statement

• The authors construct TEMPREASON by matching Wikidata knowledge-base facts with Wikipedia articles under existing public and Creative Commons licenses.They state that the dataset and source materials are intended for scientific research, while Wikipedia contexts may contain improper content not reflecting the authors’ views.Source: §9 · p. 10

A Realtime Adaptation of LMs

• The adapted TempT5 model performs competitively on RealtimeQA questions released on December 16, 2022.The evaluation uses the generation track, whose task formulation matches this paper’s OBQA setting.Source: A Realtime Adaptation of LMs · p. 12

• Temporal span extraction pre-training initializes T5-base and runs for 100K steps with batch size 8, learning rate 2e-5, and maximum input length 512.Source: B Implementation Details · p. 12
• Task-specific fine-tuning uses three epochs, while maximum input lengths are 128 for CBQA, 512 for ReasonQA, and 1,024 for OBQA.The final checkpoint is used for evaluation.Source: B Implementation Details · p. 12
• Time-sensitive reinforcement learning follows PPO with a reward function rather than a reward model, using initial KL penalty 0.05, target KL coefficient 6, and γ=0.99.Source: B Implementation Details · p. 12

C Comparison of TEMPREASON and Prior Datasets

• TEMPREASON is the first compared dataset to include all three temporal reasoning types and the ReasonQA setting.The comparison is presented against prior time-sensitive question-answering datasets in Table 11.Source: C Comparison of TEMPREASON and Prior… · p. 12, Table 11 · p. 13

D TEMPREASON Templates

• The templates convert Wikidata facts into natural questions, sampling L2 times between fact start and end dates and using month-year formatting.The templates also define object-entity placement and report query counts for L2 and L3 training sets.Source: D TEMPREASON Templates · p. 12, Table 13 · p. 14