Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs

1 Introduction

• The arithmetic/semantics distinction is widely used to organize benchmark design, model training, and performance reporting, but its cognitive validity remains unclear.Temporal arithmetic involves calculations over times and dates, while temporal semantics concerns before–after relations and temporal causality.Source: §1 · p. 1
• Benchmark tasks may reward numerical calculation, language comprehension, or memorized world knowledge rather than the temporal ability their labels imply.The authors frame this as a construct-validity concern: task surface form may not identify the abilities driving performance.Source: §1 · p. 1
• Same-mode fine-tuning gains remain highly localized to training data and barely generalize to unseen datasets, challenging the labels as indicators of shared abilities.The authors test transfer within and across modes as one probe of whether each mode reflects stable capabilities.Source: §1 · p. 1
• Across 18 cognitive dimensions, arithmetic and semantic datasets show no clear separation, and high quantitative- or logical-reasoning training does not yield larger gains in the corresponding mode.Instead, task performance depends more strongly on knowledge demands.Source: §1 · p. 2
• The results suggest temporal reasoning benchmarks are better characterized as knowledge-heavy tasks than as tests of distinct arithmetic or semantic capabilities.Performance depends more on learned regularities and background knowledge than on explicit reasoning over temporal quantities or relations.Source: §1 · p. 2

2 Background

• Existing temporal benchmarks treat arithmetic and semantics as meaningful categories, yet none tests whether they reflect separable underlying abilities.Examples include TRAM’s math-related tasks, TimeBench’s symbolic versus semantic tasks, and ToT’s Arithmetic and Semantic subsets.Source: §2 · p. 2
• Temporal benchmark categories may not measure their named abilities, and the main performance bottleneck can diverge from a task’s stated capability as tasks become more complex.The paper applies this construct-validity concern to whether temporal arithmetic/semantic labels predict transfer and cognitive abilities.Source: §2 · p. 2
• Within-mode transfer is a probe of whether benchmark labels track shared abilities, since narrow-domain training can generalize when tasks draw on the same capability.Prior work reports transfer across mathematical tasks, while temporal fine-tuning work had not tested cross-mode transfer.Source: §2 · p. 2

3.1 Datasets

• Nine temporal datasets from eight benchmarks are used, with ToT Arithmetic and Semantic counted separately and TCP further split into Long and Short for demand-profile analyses and item-level experiments.The datasets vary in temporal phenomena, task formats, and construction methods; training, validation, and test splits are reported.Source: §3.1 · p. 2, Table 1 · p. 3

3.1.1 Cognitive capability labelling

• Each item receives a DeLeAn annotation vector whose dimensions score cognitive demand, including knowledge, metacognition, attention, comprehension, QLq, and QLl.The study relabels data with Gemini 3.1 Flash Lite and compares its labels with GPT-4o’s to check reliability.Source: §3.1.1 · p. 3

3.1.2 Operationalising the arithmetic/semantic distinction

• Benchmark-level partitions use ToT’s explicit Arithmetic/Semantic split and Su et al.’s classification of TRAM subtasks; item-level partitions filter for QLq greater than QLl or vice versa, yielding six training conditions including two zero-demand controls.QLq dominates only 15 of 8,530 training items (1,288 including ties), while QLl dominates 1 item (771 including ties).Source: §3.1.2 · pp. 3–4, Table 2 · p. 4

3.1.3 Non-temporal QL-heavy training

• Non-temporal QL-heavy training uses 428 examples selected for QL demand of at least 3 and dominance over other dimensions, with mean QL demand 4.13.The examples are drawn from six named non-temporal datasets and benchmarks.Source: §3.1.3 · p. 4

3.2 Evaluation

• Evaluation uses accuracy and normalizes temporal expressions so equivalent date formats are treated alike; dimension-specific ability is estimated from dominant-demand slices using fitted logistic curves and AUC.Four cognitive dimensions are excluded from curve analysis because their coverage is too narrow or sparse for reliable estimates.Source: §3.2 · p. 4

3.3 Fine-tuning

• Fine-tuning transfer tests whether training on one mode improves performance on the corresponding mode, using GRPO with a fixed budget of 400 optimization steps and a batch size of 16.The study uses six models from three families and two size ranges; reward settings include accuracy and one-minus-accuracy, with QL-scaled rewards for item-level experiments.Source: §3.3 · pp. 4–5

4.1 Cognitive demand profiles do not reflect the temporal reasoning dichotomy

• Across items, QLq and QLl demands cluster near the diagonal (modal level 1 each); ToT Semantic has 679 items with QLq ≥3 versus 41 in Arithmetic.Two Semantic question types requiring arithmetic partly explain this contrast.Source: Figure 1 · p. 5, §4.1 · p. 5
• DeLeAn PCA separates high-demand TCP and ToT Semantic clusters, but the remaining benchmarks show no mode-consistent grouping.PC1 and PC2 explain 40.5% and 15.5% of variance, respectively; the leading structure reflects overall demand.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 Performance gains are dataset-specific and do not transfer across modes

• The exploratory Qwen3-32B run showed smaller, mixed transfer effects, including a 3.7-point TRAM Arithmetic-to-ToT Arithmetic gain, without a consistent mode-specific pattern.Across the six main models, the dataset-specific pattern was consistent and not driven by individual outliers.Source: §4.2 · p. 6
• Off-diagonal transfer was essentially absent; TRAM exceptions (11.1 pp arithmetic-to-semantic; 3.4 pp semantic-to-arithmetic) largely reflect shared answer formatting.TRAM’s two modes share a prompt template and multiple-choice answer space; ToT’s modes differ in content and answer format.Source: §4.2 · p. 6, Figure 3 · p. 7
• One-minus-accuracy degradation spread more broadly off diagonal, with the largest drops for ToT arithmetic (−26 pp) and ToT Semantic (−22.8 pp).For TRAM, the asymmetry between arithmetic-to-semantics and reverse transfer under accuracy training reverses under unlearning.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 Quantitative and logical reasoning are not the main drivers for performance

• QLl=QLq+1 training produced the largest gains for both dimensions; the +2 condition was weaker, possibly because it contained only 54 items.QLq-dominant training did not preferentially improve QLq; the authors note that QLq demands are generally lower than QLl demands.Source: §4.3 · p. 7
• QL-dominant non-temporal training raised QLl and QLq by 0.15 AUC each, while temporal-mix accuracy changed by mean ∆= −0.06 pp.The temporal accuracy range was −1.23 to +0.55 pp.Source: §4.3 · p. 7
• Across training conditions, gains in KNs, KNc, and CEc exceed QLl and QLq gains, even when data emphasize quantitative or logical demand.The reverse one-minus-accuracy results also show the largest capability degradation for KNs.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

• Cognitive-demand analyses and fine-tuning experiments find that arithmetic and semantic benchmark items do not reflect distinct reasoning abilities, and same-mode gains do not transfer across benchmarks.Source: §5 · p. 8
• Knowledge dimensions improve more than QLq or QLl, while specialized QL-heavy training raises QL capability without improving either temporal mode.Source: §5 · p. 8
• The authors conclude that current benchmarks largely measure retrieval and application of time-related regularities, and recommend controlling knowledge demands and validating benchmark categories through capability-based analyses.Source: §5 · p. 8

6 Limitations

• Experiments use models of up to 12 billion parameters, and the transfer analysis is restricted to ToT and TRAM, the only benchmarks with explicit or reconstructable arithmetic/semantic partitions.The authors note one larger-model instance with about 3 percentage points of arithmetic-mode transfer and say future work could test larger models and additional benchmarks.Source: §6 · p. 9

A Demand level annotations (DeLeAn)

• The study uses DeLeAn demand vectors as structured, interpretable item-level features; its rubrics include cognitive, knowledge-related, and extraneous dimensions, but not unguessability.Each of 18 dimensions has an open-scale score, with 0 indicating no meaningful demand and 5+ indicating demand beyond the rubric’s highest anchor.Source: A Demand level annotations (DeLeAn) · p. 13

B.1 Benchmark sizes

• The demand-profile analysis labels nine datasets as ten subsets, keeping TCP Long and Short separate; TRAM is subsampled so no benchmark dominates training conditions.The study uses existing mode labels for ToT and TRAM in transfer experiments and DeLeAn profiles for the ten subsets.Source: §B.1 · p. 13

B.2 Partitioning TRAM into arithmetic and semantic tasks

• TRAM items show weak separation by assigned mode in the demand-profile PCA, consistent with arithmetic and semantic tasks not being strongly distinguishable by demand vectors.The TRAM partition uses four math-time subtasks for arithmetic and four selected pure-time subtasks for semantics.Source: §B.2 · p. 13

B.3 Cognitive demand profiles of ToT Semantic question types

• ToT Semantic question types have heterogeneous cognitive profiles: RelationDuration and Timeline lean slightly more quantitative, while FirstLast, BeforeAfter, and EventAtTimeOfOtherEvent show the largest negative QLq−QLl differences.This heterogeneity supports the finding that the semantic label subsumes tasks with meaningfully different cognitive profiles.Source: §B.3 · pp. 13, 15

B.4 Per-benchmark cognitive demand profiles

• QLq and QLl are not the highest-demand dimensions in any subset; knowledge dimensions including KNc and CEc dominate, while TCP Short and Long are demanding across nearly all dimensions.Date Arithmetic has near-zero demand on most dimensions except QLq and KNc; most subsets concentrate QLq and QLl demands at levels 0–2.Source: §B.4 · p. 15

C Metrics

• Evaluation generally extracts and canonicalizes answers before comparing normalized strings, structured objects, or sets, with procedures varying by benchmark.TRAM uses the final option letter from its expected answer template and exact-match accuracy.Source: C Metrics · p. 15, §C.1 · p. 15

C.2 TCP accuracy

• TCP predictions are extracted from boxed LaTeX and compared by exact string match, with “GMT” removed only for TCP Short.The system prompt specifies the expected answer format.Source: §C.2 · pp. 15–16

C.3 Test of Time

• ToT scoring extracts the first valid JSON object and compares answer-bearing content after subset-specific normalization.Arithmetic also normalizes schema and type mismatches and sorts unordered lists; Semantic converts prediction and reference to strings before exact comparison.Source: §C.3 · p. 16

C.4 TimeBench derived benchmarks

• TimeBench-derived answers are extracted after a fixed marker and scored with task-specific exact-match or F1 procedures, including month-year comparison for Date Arithmetic.TempReason, TimeQA, and MenatQA use SQuAD exact match and F1; TimeDial uses set exact match and set-based F1.Source: §C.4 · pp. 16, 18

C.5 Non-temporal multiple-choice accuracy

• For single-answer multiple-choice examples, the evaluator extracts an answer-bearing option letter from the completion’s final 300 characters and compares it with the gold label.If no answer-bearing expression is found, it falls back to the final isolated letter in that range.Source: §C.5 · p. 18

C.6 TruthQuest accuracy

• TruthQuest scores predictions as sets of knight-option labels and awards a match only when the predicted and reference sets are identical.Ordering and duplicate labels are ignored.Source: §C.6 · p. 18

C.7 Math and OmniMath accuracy

• Math and OmniMath scoring depends on answer type: integer answers are compared exactly, sequence answers use order-sensitive JSON-array comparison, and fractions are normalized as exact rationals.Multiple-choice items use the option-letter procedure.Source: §C.7 · p. 18

C.8 ChemLLMBench and Reasoning accuracy

• ChemLLMBench predictions are parsed into canonical isomeric SMILES for exact comparison, while free-text Reasoning answers are normalized and matched as substrings.Chemistry predictions are searched from right to left for a parseable SMILES-like token.Source: §C.8 · p. 18

D Training settings

• Training used Transformers, TRL, and PEFT, with LoRA rank 16 and scaling parameter α = 32; runs used configurations ranging from single-node training to three-node Qwen3-32B training.Full runs generally took several hours, with longer runs approaching or exceeding 12 hours.Source: D Training settings · pp. 18–19

E Label reliability

• The reliability comparison reports within-group interrater reliability and supplements it with exact matches, within-one matches, and Krippendorff’s alpha.The study also compares mean demand levels between Gemini and GPT.Source: E Label reliability · p. 19
• Overall rWG agreement is close to 1, but KNc agreement is 0.66; CEc has lower Krippendorff’s alpha because both annotators mostly assign levels 1–2, increasing by-chance agreement.The analysis clips item-level rWG to [0,1] and averages items with both labels present in each dimension.Source: E Label reliability · p. 19
• GPT-4o assigns KNc demands an average 1.59 points higher than Gemini, which treats supplied answer information as requiring little or no prior everyday knowledge.GPT-4o appears to conflate topical complexity with everyday knowledge, which the rubric excludes when the content is specialized academic knowledge.Source: E Label reliability · p. 19
• With Gemini 3.1 Flash Lite, knowledge dimensions still show the largest capability increases across training settings, while QLl and QLq barely improve; CEc does not reproduce consistently.KNs and, in one instance, KNf show the largest increases instead of KNc.Source: E Label reliability · p. 19

F Per-model results

• All six models improved on their own training sets under the accuracy reward (6/0 in every diagonal cell).Source: F Per-model results · p. 19
• Tables 8 and 9 cover inter-rater agreement and mean demand levels for Gemini 3.1 Flash Lite and GPT-4o annotations.Source: Table 8 · p. 20, Table 9 · p. 20
• Figure 8 reports average absolute AUC changes across six models from three families and two sizes for accuracy and one-minus-accuracy reward settings.Source: Figure 8 · p. 21, F Per-model results · p. 21
• Off-diagonal transfer was not driven by outliers, while one-minus-accuracy training degraded all six models on ToT Arithmetic and five on TRAM Arithmetic.Most off-diagonal cells had near-even improve/worsen counts; TRAM Arithmetic baseline accuracy ranged from 25.74 to 87.74.Source: F Per-model results · p. 21

G Non-temporal QL-heavy training

• QLq and QLl had the largest gains across 14 dimensions, at +0.15 AUC each, after training on non-temporal QL-dominant data.Source: G Non-temporal QL-heavy training · p. 21
• Temporal-mixture accuracy was essentially unchanged after training, with a mean ∆ of −0.06 pp and per-model range −1.23 to +0.55 pp.Source: G Non-temporal QL-heavy training · p. 21
• The training increased the targeted capabilities without translating into better performance on the evaluated temporal benchmarks.Source: G Non-temporal QL-heavy training · p. 21

H Larger models

• Computational cost of GRPO prevented the full experimental suite on larger models, with some ToT prompts spanning several thousand tokens.Source: H Larger models · p. 21
• Table 22 disaggregates transfer results by train–test combination and reward setting, including baseline accuracy ranges, performance deltas, and counts of models improving or worsening.Source: Table 22 · p. 28
• Training on TRAM Arithmetic improved ToT Arithmetic by 3.7 percentage points and ToT Semantic by 1.9 points; TRAM Semantic training improved ToT Semantic by 1.9 points and TRAM Arithmetic by 2.7 points.Source: H Larger models · p. 28
• The results offer tentative evidence of arithmetic transfer, not a consistent mode-specific pattern, and one larger model leaves scale effects versus ordinary variation unresolved.Source: H Larger models · p. 28
• Training Qwen3-32B on TRAM subsets produced accuracy changes from −0.2 to +3.9 points, below the +6.8 to +23.1-point in-domain gains.Figure 9 reports changes in accuracy for the trained variants.Source: H Larger models · p. 28, Figure 9 · p. 30