PaperlayerView exact revision on arXiv

Source-linked AI summary

TiMoE: Time-Aware Mixture of Language Experts

Robin Faro, Dongyang Fan, Tamar Alphaidze, Martin Jaggi

arXiv:2508.08827v1cs.CL

TL;DR

LLMs can be temporally stale and leak future information because standard pretraining conflates knowledge from different periods. TiMoE trains isolated time-specific experts and causally combines those available by the query timestamp, while TSQA measures temporal errors. The co-adapted variant matches or exceeds the best single-period expert and reduces temporal-drift errors by up to 15%.

  • Problem

    Standard LLM pretraining conflates temporal contexts, making it difficult to distinguish past, present, and future information and weakening time-consistent prediction.

    Source: §1 · p. 1

  • Method

    TiMoE pretrains separate experts on non-overlapping time slices and aggregates only experts whose training data does not exceed the query timestamp.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    15% reduction in temporal-drift errors is reported for TIMOE-COADAPT on TSQA, while it matches or exceeds the best single-period expert across eight standard NLP tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Causal routing with modular time-segmented experts provides a recipe for chronologically grounded language models while retaining broad multi-period knowledge.

    Source: §5 · pp. 8–9

    Corpus slices are dated by document collection rather than original creation, and limited tokens per slice can underrepresent present-day knowledge.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are typically trained on fixed snapshots of the web, which means that their knowledge becomes stale and their predictions risk temporal leakage: relying on information that lies in the future relative to a query. We tackle this problem by pre-training from scratch a set of GPT-style experts on disjoint two-year slices of a 2013-2024 corpus and combining them through TiMoE, a Time-aware Mixture of Language Experts. At inference time, TiMoE masks all experts whose training window ends after the query timestamp and merges the remaining log-probabilities in a shared space, guaranteeing strict causal validity while retaining the breadth of multi-period knowledge. We also release TSQA, a 10k-question benchmark whose alternatives are explicitly labelled as past, future or irrelevant, allowing fine-grained measurement of temporal hallucinations. Experiments on eight standard NLP tasks plus TSQA show that a co-adapted TiMoE variant matches or exceeds the best single-period expert and cuts future-knowledge errors by up to 15%. Our results demonstrate that modular, time-segmented pre-training paired with causal routing is a simple yet effective path toward LLMs that stay chronologically grounded without sacrificing general performance much. We open source our code at TiMoE (Github): https://github.com/epfml/TiMoE

1 Introduction

TiMoE addresses temporal unawareness in LLMs by isolating training periods and causally aggregating only knowledge available at a query’s timestamp. The paper introduces TSQA and reports improved temporal consistency alongside broad task performance.

  • A causal timestamp mask activates experts trained no later than the query, allowing historical knowledge while excluding future information.The approach aggregates eligible experts rather than restricting inference to a single period.Source: §1 · p. 2
  • The time-sensitive pipeline also supports analyzing how language use, entity relevance, and factual content change across periods.Source: §1 · p. 2
  • TiMoE trains separate time-specific experts on isolated periods, eliminating future leakage by design.Past experts remain frozen while new experts can be added for later periods, supporting modular updates without reprocessing the full corpus.Source: §1 · pp. 1–2
  • TSQA is a 10k-question benchmark that labels alternatives as past, future, or irrelevant for fine-grained temporal-hallucination diagnosis.Source: §1 · p. 2
  • 15% reduction in temporal-drift errors is reported for TIMOE-COADAPT on TSQA, while matching or exceeding the best single-period expert across eight standard NLP tasks.Source: §1 · p. 2

2 Related Work

Prior work addresses temporal knowledge through timestamp conditioning, continual or adapter-based adaptation, and time-sensitive benchmarks, while model-merging methods provide complementary combination strategies. TiMoE differs by pretraining isolated experts from scratch to prevent cross-period leakage.

  • Timestamp-conditioned, continual-pretraining, and era-specific adapter methods improve temporal representations or target-period performance, but adapters may retain cross-period knowledge.Source: §2 · pp. 2–3
  • TiMoE performs time-aware pretraining from scratch with strict temporal isolation, emphasizing leakage prevention alongside expert routing.Source: §2 · p. 3
  • Existing benchmarks test changing facts, dated recall, temporal reasoning, and semantic ordering; the paper adds labels for current, past, future, or irrelevant answers.Source: §2 · p. 3
  • Model soups, parameter merging, adapter fusion, and LoRA composition establish precedents for combining model components or experts.Source: §2 · p. 3

3 Our Design

TiMoE uses two stages: independently trained experts on non-overlapping temporal slices, followed by timestamp-aware aggregation of eligible experts. Aggregation combines log-probabilities under causal masking with fixed or learned routing.

3 Our Design

  • The design divides pretraining into time-specific expert learning and time-aware knowledge aggregation.Source: §3 · p. 3

3.1 Stage 1: Time-specific LM experts Learning

  • Six experts cover 2013–2024 in consecutive non-overlapping two-year intervals, balancing temporal resolution with data availability.Source: §3.1 · pp. 3–4

3.2 Stage 2: Time-aware aggregation of LM experts

  • For query timestamp tq, TiMoE excludes experts trained on future data and aggregates eligible experts’ predictions in log-probability space.Only experts with training periods k ≤ tq participate, preserving causal validity while accumulating historical knowledge.Source: §3.2 · p. 4
  • Aggregation weights are non-negative and sum to 1, with equal-weight averaging providing a training-free ensemble of past experts.Source: §3.2 · p. 4
  • TiMoE-LearnedAvg trains a router while freezing experts, whereas TiMoE-CoAdapt jointly updates the router and the expert corresponding to each document’s time window.Source: §3.2 · p. 4
  • TiMoE-Year is a hard-routing baseline that selects the expert whose training window matches the query year.Source: §3.2 · p. 4

4 Experiments

The experiments evaluate TiMoE’s time-segmented experts, TSQA benchmark, general-purpose performance, temporal awareness, and semantic-change analysis. TiMoE improves time-sensitive modeling while incurring a measurable general-performance trade-off against GPT2-FULL.

4.1 Experimental setup

  • The experimental setup trains GPT-2-medium experts on six two-year bins from a 2013–2024 corpus and uses timestamp-conditioned expert aggregation.Experts use 480M parameters and 1024-token contexts; router-based aggregation applies a time-dependent mask.Source: §4.1 · p. 5

4.2 A new curated benchmark for time sensitive knowledge

  • TSQA contains 10,063 time-sensitive multiple-choice questions spanning 2013–2024, with distractors drawn from past, future, or irrelevant contexts.Questions have four candidate answers, with one valid for the target year; construction uses timelines from three existing datasets and LLM-generated alternatives.Source: §4.2 · pp. 5–6

4.3 Comparable performance on general-purpose benchmarks

  • TIMOE-COADAPT underperforms GPT2-FULL by an average of 5.6% across general-purpose tasks, while outperforming average and best single-expert modular baselines across most tasks.GPT2-FULL is trained on the entire 100B-token corpus; Table 1 compares modular aggregation strategies across eight benchmarks.Source: §4.3 · p. 6, Table 1 · p. 6

4.4 Improved time awareness through time-sensitive modeling

  • TiMoE variants show approximately 3% higher TSQA performance than comparable frontier models despite using 480-million-parameter experts versus 3-billion-parameter baselines.The timestamp is provided explicitly to TiMoE models for masking future experts, whereas non-time-aware baselines receive temporal information in the prompt.Source: §4.4 · p. 7, Figure 3 · p. 7
  • TiMoE reduces past and irrelevant responses and consistently outperforms baselines across the evaluated years on TSQA.These findings are presented through answer-type distributions and yearly accuracy plots.Source: §4.4 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8

4.5 Cross-temporal semantic change discovery

  • The semantic distance between “coronavirus” and pandemic-related terms is smallest in 2019–2020 and increases by 2022–2023, reflecting changing contextual associations.Time-specific embeddings are extracted from final hidden-layer tokens and compared using cosine similarity.Source: §4.5 · p. 8, Figure 6 · p. 8

5 Conclusion

TiMoE combines strictly time-segmented GPT-2 experts with causal routing to keep predictions chronologically grounded. It reduces temporally inconsistent answers on TSQA while preserving steadier performance across years, with a manageable general-performance cost.

5 Conclusion

  • TiMoE cuts temporally inconsistent answers on the 10k-question TSQA benchmark by roughly 15% while delivering steadier accuracy across years.The system masks experts trained after the query year and blends earlier specialists.Source: §5 · pp. 8–9
  • On eight standard NLP benchmarks, TiMoE models slightly underperform standard GPT2, indicating a manageable cost of time-awareness rather than a fundamental barrier.The conclusion frames the approach as a modular recipe for temporally causal and evolving language models.Source: §5 · p. 9

6 Limitations and Future Work

The corpus is partitioned by document collection dates rather than original creation dates, and limited tokens per slice can underrepresent present-day knowledge. Future work proposes finer temporal partitions, larger base models, and creation-date partitioning.

Limitations

  • Collection dates rather than original creation dates determine the corpus partitions, so individual slices may contain content originating from earlier periods.Source: §6 · p. 9
  • Limited tokens per slice can underrepresent present-day knowledge and make time-sensitive evaluation more challenging.Source: §6 · p. 9

Future Work

  • Future work includes finer-grained temporal partitioning, scaling to larger base models, and partitioning by document creation dates.Source: §6 · p. 9

Prompt Template for Time-Sensitive QA Generation

The template supplies a question, year of interest, and possible answers across years, then constructs four options with one correct answer and three timeline-based wrong answers. Each option receives a correct, past, future, or irrelevant tag in a strict dictionary format.

Inputs

Task Instructions

Output Format

Example

Related papers

Loading 2508.08827v1…