PaperlayerView exact revision on arXiv

Source-linked AI summary

What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?

Gagan Bhatia, Ahmad Muhammad Isa, Maxime Peyrard, Wei Zhao

arXiv:2603.19017v1cs.CLcs.AI

TL;DR

The paper asks whether multilingual temporal reasoning is primarily controlled by date tokenisation or by internal representations of time. It introduces MultiTempBench, a controlled multilingual benchmark with fragmentation and geometric-probing analyses, and finds that the stronger predictor depends on language-resource level: temporal linearity in high-resource languages and fragmentation in low-resource languages.

  • Problem

    Existing temporal benchmarks largely emphasize English and Gregorian forms, while multilingual temporal expressions, calendars, and tokenisation effects remain insufficiently characterized.

  • Method

    The paper builds a 15,000-example benchmark across five languages, three temporal tasks, and multiple calendars, then measures date fragmentation with mDFR and probes temporal representation geometry.

  • Results

    Temporal linearity is the strongest predictor of temporal reasoning in high-resource languages, whereas date fragmentation is the stronger predictor in low-resource languages.

  • Takeaways & Limitations

    Tokenisation controls access to calendar components in low-resource settings, while internal temporal geometry matters more in high-resource settings.

  • Takeaways & Limitations

    The controlled benchmark covers only five languages, three task families, selected calendars, translated and templated data, and zero-shot direct-answer evaluation.

Abstract

from arXiv · show

We present MultiTempBench, a multilingual temporal reasoning benchmark spanning three tasks, date arithmetic, time zone conversion, and temporal relation extraction across five languages (English, German, Chinese, Arabic, and Hausa) and multiple calendar conventions (Gregorian, Hijri, and Chinese Lunar). MultiTempBench contains $15,000$ examples built by translating $750$ curated English questions and expanding each into controlled date-format variants. We evaluate 20 LLMs and introduce the multilingual Date Fragmentation Ratio (mDFR), calibrated with human severity ratings, together with geometric-probing analyses of internal temporal representations. We find tokenisation quality of temporal artefacts is a resource-dependent bottleneck: in low-resource languages and rarer calendar formats, fragmentation disrupts Year/Month/Day separation and accuracy collapses, while high-resource settings are often robust to digit-level splitting. Beyond tokenisation, crossed mixed-effects regression shows that temporal linearity is the strongest predictor of temporal reasoning in high-resource languages, whereas fragmentation is the stronger predictor in low-resource languages. Code is available at: https://github.com/gagan3012/mtb

1 Introduction

The paper frames multilingual temporal reasoning as a joint problem of heterogeneous temporal expressions, tokenisation, and internal time representations. It introduces a controlled benchmark and argues that the dominant bottleneck shifts with language-resource level.

  • Existing temporal benchmarks overwhelmingly focus on English and Gregorian representations, leaving multilingual and cross-calendar reasoning less characterized.
  • Temporal tokenisers can fragment structured dates, erasing year/month/day boundaries and complicating multi-digit arithmetic.Low-resource languages often experience heavier fragmentation, increasing sequence length and creating unstable grouping boundaries.
  • MULTITEMPBENCH contains 15,000 examples across 5 languages, 3 temporal tasks, multiple date-format complexities, and Gregorian, Hijri, and Lunar calendars.The benchmark evaluates open-weight and proprietary LLMs in a zero-shot setting while varying language resource level, format complexity, and calendar system.
  • The paper introduces mDFR, which weights semantically destructive date segmentations using human severity ratings.
  • In low-resource languages, date fragmentation is the stronger predictor of failure, whereas temporal linearity is the strongest predictor in high-resource languages.The authors describe a two-stage view: tokenisation affects access to calendar components, while temporal linearity reflects internal temporal representations.

2 Related Works

Related work identifies tokenisation bias and a distinction between memorisation and reasoning in temporal tasks. This paper extends those lines by using controlled date expressions to separate tokenisation effects from temporal reasoning.

  • Low-resource languages and dialects often incur a token tax through inflated sequence lengths, which can degrade performance and increase compute costs.The related work also links fragmentation of morphological units to possible effects on numeric and temporal reasoning.
  • MULTITEMPBENCH systematically isolates how date tokenisation quality affects multilingual reasoning through the mDFR metric.
  • Prior research distinguishes stable performance on memorisation-based temporal tasks from sharp declines on reasoning-intensive tasks involving temporal shifts or new knowledge.
  • The benchmark uses controlled date expressions to disentangle tokenisation from reasoning, unlike broad benchmarks.

3 Our MULTITEMPBENCH

MULTITEMPBENCH is constructed from curated temporal questions, translated into five languages and expanded into controlled date-format and calendar variants. The resulting dataset supports multilingual evaluation across three temporal task families.

  • The benchmark derives from TRAM, ToT, and FreshBench, selecting questions with fully specified year, month, and day components.Preprocessing removes synthetic entities and internal prompting instructions to preserve grammatically correct, natural questions.
  • 750 English questions are translated into German, Chinese, Hausa, and Arabic, with two native speakers per target language validating and editing the translations.
  • Each question expands into four date-format variants per language, ranging from ISO numeric forms to localised numeric and calendar-specific phrases.Calendar-specific variants use Hijri and Chinese Lunar conversions verified by native speakers.
  • The expansion produces 3,000 questions per language, totalling 15,000 questions.

4 Our Approach

The approach measures multilingual date-tokenisation quality with mDFR and probes whether temporal values form structured, linearly decodable representations in model embeddings.

  • Multilingual Date Fragmentation Ratio: mDFR measures date-tokenisation quality by penalising destructive splitting, lost delimiters, token-count inflation, and structural divergence.Its coefficients are calibrated against human judgements of fragmentation severity across target languages.
  • Temporal Geometry: The study extracts hidden representations for years across five languages and three date formats using multiple full-date samples per year.The sampled dates are embedded in declarative templates, and the final-token hidden states are collected for analysis.
  • Temporal Geometry: Successive year representations are analysed through vector differences, paths of line segments, and an overall path direction.These geometric quantities describe how year representations change through embedding space.
  • Temporal Geometry: Temporal linearity tests whether calendar values form a one-dimensional structure in embedding space.The analysis applies the same idea to Year, Month, and Day components.
  • Temporal Geometry: A linear regressor decodes calendar components from averaged hidden representations, with R2 measuring the quality of recovery.Higher R2 indicates that the component is organised along an approximately ordered axis in the embedding space.

5 Experiments

Experiments compare diverse decoder-only LLMs, their native tokenisers, and a linguistically informed semantic baseline under zero-shot evaluation. The setup examines how model and tokenisation choices affect multilingual temporal reasoning, including fragmentation patterns across resource levels.

  • Models: The evaluation covers Qwen3, LLaMA 3, OLMo, Gemma, Mistral, Phi-4, GPT-4o, and GPT-4o-mini models.The open models span different sizes, including Qwen3 from 0.6B to 14B parameters and LLaMA 3 at 8B and 70B.
  • Tokenisation Baseline: The native subword tokeniser is compared with a deterministic baseline that segments dates into semantic primitives while preserving delimiters and whitespace.The baseline is language-aware, including Arabic-Indic numerals, Chinese temporal markers, and Hijri suffixes.
  • Tokenisation Analysis: mDFR quantifies tokenisation quality, weighting structural divergence and root splitting more heavily than simple token-count inflation.The learned coefficients are α = (0.2, 0.2, 0.1, 0.5).
  • Tokenisation Analysis: Hausa exhibits the highest reported DFR of 0.78, with both numeric splitting and semantic fragmentation of the month name.The example Oktoba 10, 2034 is segmented as O|kt|oba alongside numeric splitting.
  • Evaluation: Models are evaluated zero-shot without fine-tuning, chain-of-thought demonstrations, or external knowledge, and predictions are judged against gold-standard aliases.GPT-4o classifies outputs as CORRECT, INCORRECT, or NOT_ATTEMPTED.

6 Results

Results across three temporal tasks show a resource-dependent shift in what limits multilingual temporal reasoning: date fragmentation is most consequential in low-resource languages, whereas temporal linearity better predicts performance in high-resource languages.

  • Multilingual Performance: Temporal reasoning performance is stronger in English, Chinese, and German but degrades sharply in Hausa, indicating a brittle low-resource regime.Gemma 3 reaches 59.2% average accuracy, compared with 57.3% for Llama 3.1 and 20.0% for GPT-OSS.
  • Date Fragmentation: Higher mDFR predicts lower date-arithmetic accuracy in low-resource languages, with correlations of r = −0.97 in Hausa and r = −0.89 in Arabic.The relationship is much weaker in German, Chinese, and English, where fragmentation does not produce a comparable accuracy collapse.
  • Temporal Geometry: Temporal linearity correlates strongly with accuracy in English (r = 0.77) and Chinese (r = 0.75), but weakly in Hausa (r = 0.10).The analysis measures linear-probe R2 for hidden-state representations of Day, Month, and Year.
  • Temporal Geometry: Year linearity is generally the most predictive component, especially in English and Chinese, consistent with Year providing an ordered backbone for temporal operations.Figure 4 compares Day, Month, and Year probe R2 correlations within each language.
  • Mechanism Comparison: A crossed mixed-effects regression over 285000 predictions found a significant mDFR × linearity × resource interaction (β = 0.016, p = 0.021).In low-resource languages, mDFR is the stronger predictor (β = −0.126, p < 0.001); in high-resource languages, linearity is stronger (β = 0.087, p < 0.001), while mDFR has only a weak effect (β = 0.009, p = 0.056).

7 Conclusion

MULTITEMPBENCH shows that multilingual temporal intelligence requires more than vocabulary expansion: temporal information must remain accessible and computable internally. The dominant bottleneck depends on language-resource regime, with fragmentation stronger in low-resource settings and temporal linearity stronger in high-resource settings.

  • Temporal intelligence requires making temporal information accessible and computable in the model’s internal space, not merely adding vocabulary.
  • In low-resource regimes, date fragmentation is the stronger predictor of temporal-reasoning failure.
  • In high-resource regimes, temporal linearity is the stronger predictor of performance.

Limitations

The benchmark is a controlled diagnostic whose coverage, construction, evaluation setting, and correlational probing limit how broadly its findings should be generalized.

  • The benchmark covers five languages and three task families, so it does not fully represent multilingual temporal phenomena.It omits additional scripts and dialects, code-mixing or noisy text, domain-specific jargon, and other calendar conventions.
  • Translation and templated format variation from curated English seeds isolate surface-form effects but may under-sample natural expression distributions and errors.
  • Zero-shot direct answering improves comparability but may understate performance under tools, prompting, or fine-tuning.LLM-based output normalization also introduces residual noise from judge mistakes and format ambiguity.
  • Associations between fragmentation, temporal linearity, and performance remain correlational and probe-centric.

Ethical Considerations

The paper frames multilingual tokenisation disparities as model and data-coverage properties rather than inherent language deficits, while emphasizing validation and governance for high-stakes use.

  • Tokenisation disparities should be framed as properties of model design and data coverage, not inherent deficits of particular languages.
  • Calendar conversion and formatting errors can have real consequences in downstream high-stakes contexts.Users should document conversion assumptions and seek native-speaker and domain-expert review when decisions matter.
  • Extensions should avoid identifiable or sensitive information, and releases should document third-party API versions and settings where applicable.Open alternatives should be provided where feasible to improve reproducibility and address governance concerns.

Broader Impact

The benchmark can help audit and improve multilingual temporal reasoning across scripts, languages, and calendar conventions. Its analyses may also guide tokeniser and training-data interventions, although leaderboard use can distort incentives.

  • The benchmark can help the community audit and improve temporal reasoning across scripts, languages, and calendar conventions.
  • Linking surface segmentation properties to downstream competence may guide more principled tokeniser and training-data interventions.
  • Treating the benchmark as a leaderboard target can distort incentives and encourage narrow optimization.

A Appendix

The appendix describes MultiTempBench’s multilingual date-formatting pipeline and validates mDFR against human judgments. Results show fragmentation is more damaging in low-resource languages, while higher-resource settings are generally more robust.

  • Benchmark construction: The unified pipeline standardizes dates into Year, Month, Day before applying language-specific formatters for English, Arabic, Chinese, Hausa, and German.It supports ISO, slash, long, and calendar-oriented variants, including Hijri and Lunar conversions.
  • mDFR validation: mDFR measures tokenisation deviation from semantic segmentation using root splitting, delimiter loss, token-count increase, and distributional divergence.Its score F ∈[0, 1] combines four weighted error components.
  • mDFR validation: ρ = 0.89 links mDFR to human fragmentation-severity ratings, compared with BLEU at ρ = 0.43.This supports mDFR as a closer measure of perceived semantic disruption than the general-purpose comparison metric.
  • mDFR validation: Distributional Divergence (θ) is the dominant learned severity factor, followed by structural breaks, while token-count inflation has a minor role.Human-derived weights were reported as similar to the normalized original intuitive weights.
  • Task correlations: Temporal relation extraction shows stronger fragmentation penalties in Hausa (r = −0.58) and Arabic (r = −0.29) than in English (r = 0.06), German (r = 0.28), or Chinese (r = 0.08).The reported association is between higher fragmentation and lower accuracy in the two low-resource languages, especially Hausa.
  • Task correlations: Time zone conversion shows negative fragmentation–accuracy correlations in Arabic (r = −0.54) and Hausa (r = −0.74), but weak relationships in English, German, and Chinese.Across the three tasks, fragmentation is most predictive of failure in low-resource languages rather than serving as a universal explanation.
Loading 2603.19017v1…