PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Language Models Consistently Encode the Current Year?

Suze van Adrichem, Aditi Bhaskar, Diyi Yang, Christopher Potts, Jing Huang

arXiv:2608.15507v1cs.CLcs.LG

TL;DR

Language models need a consistent notion of the current time for temporal reasoning, but how they represent the current year remains unclear. This paper probes associative and declarative notions of the current year and finds that they rely on different mechanisms, preventing existing methods from updating both simultaneously.

  • Problem

    How language models represent the current year is unclear, despite its importance for temporal reasoning involving time-dependent facts.

  • Method

    The paper compares associative verb-tense inference with direct declarative queries, analyzing behavior, internal representations, training distributions, and update methods.

  • Results

    Prompting shifts the declarative year with 94.6% success but shifts the associative year with only 1.7% success, while weight editing fails to generalize across both tasks.

  • Takeaways & Limitations

    The current year is not consistently encoded: the pre-training-grounded associative notion resists post-hoc modification, preventing simultaneous updating of both notions.

  • Takeaways & Limitations

    The associative inference applies specifically to prompts without embedding verbs or other constructions that shift the anchoring time away from the speaker’s present.

Abstract

from arXiv · show

A consistent concept of the current time is important for temporal reasoning, yet how language models represent the current time is not well understood. We contribute two tasks that probe the current year in conceptually distinct ways: an associative task, which infers the current year from verb tense, and a declarative task, which directly queries for the current year. Both tasks estimate current years within one year of the post-training data cutoff of instruction-tuned language models. For base models, predictions on the associative task serve as a strong proxy for the pre-training data cutoff, with an average error of only 10 months across 13 models. However, their internal mechanisms diverge: the associative task uses mechanisms similar to factual recall, while the declarative task lacks consistent causal pathways. This divergence poses a challenge for updating the current year in language models. None of prompting, SFT, or weight editing succeed in shifting the associative and declarative years simultaneously. Prompting updates the declarative year (94.6% success across 351 target years) but leaves the associative year nearly unchanged (1.7% success). Year-shifted SFT also fails to shift the associative year, matching the target year in only one of eight models. Weight editing, while effective for both tasks individually, does not generalize across both. Overall, our results show that the current year is not consistently encoded in language models: The associative notion, deeply ingrained in linguistic structures learned in pre-training, uses different causal mechanisms and resists the same modifications that easily shift the declarative notion learned in post-training.

1 Introduction

This work tests whether language models consistently represent the current year using associative and declarative tasks. Although both notions align behaviorally with training cutoffs, they rely on different mechanisms and respond differently to updates.

  • Tasks: The study introduces an associative task inferring the current year from verb tense and a declarative task directly querying it.The tasks probe the current year in conceptually distinct ways while supporting analyses of behavior, representations, and training data.
  • Findings: 0.85 years: the associative task’s average error across 13 base LMs, making it a strong proxy for the pre-training cutoff year.After SFT, models converge to within one year of their training data cutoff on both tasks; the declarative year is acquired in post-training.
  • Mechanisms: The associative current year resembles factual recall, whereas the declarative current year lacks a consistent internal mechanism.Causal interventions reveal that the two tasks’ behavioral alignment masks divergent internal mechanisms.
  • Updating: 94.6% across 351 target years: system prompting shifts the declarative current year, while the associative current year resists comparable updating.The study evaluates context specification, year-shifted fine-tuning, and low-rank weight editing as update methods.
  • Conclusion: The associative notion is pervasively encoded during pre-training and relies on causal mechanisms distinct from the post-training declarative notion.This divergence explains why existing interventions can update the declarative notion without consistently updating the associative notion.

2 Related work

Prior work studies time-dependent behavior in language models through temporal evaluation, dated-data finetuning, model steering, and prompting. Related causal work localizes and edits model concepts using interventions on attention, MLPs, representations, and feature subspaces.

  • Time-dependent behaviors in LMs: Recent studies evaluate chronological knowledge and temporal reasoning in language models, while also refreshing models with dated data and inducing alternative current times through steering or prompting.These approaches cover both benign and adversarial behaviors.
  • Localizing and editing concepts in LMs: Causal localization and editing methods include attention or MLP knockouts, representation swapping, and learned feature-subspace interventions such as distributed alignment search.The methods target mechanisms underlying concepts in language models.

3 The associative current year

The associative current year (ACY) infers an LM’s implicit time from verb-tense predictions, providing an accurate proxy for pre-training cutoff dates. Its tense associations track pre-training data and are represented through a year–tense subspace consistent with factual recall mechanisms.

  • Task and definition: ACY is the first year y where past-tense probability falls below present/future probability for prompts of the form “In {year} there.”The task uses was/were as past forms and is/are/will as present/future forms, which together account for over 90% of next-token predictions.
  • Accuracy as a cutoff proxy: Neither event-specific nor current-knowledge factual probes recovers the current year cleanly, making verb tense the more reliable and scalable signal.This supports distinguishing associative tense-based recovery from direct factual knowledge probes.
  • Accuracy as a cutoff proxy: 0.85 years (≈10 months) is the ACY’s average error against pre-training cutoff dates across 13 base LMs, versus 2.71 years for perplexity predictions.ACYM performs slightly better at 0.75 years (≈9 months), though OLMo2-7B contributes over half of its error with a five-year deviation.
  • Relationship to pre-training data: Both OLMo2 models and both count models shift ACY at the pre-training stage boundary, from ≤2022 in stage 1 to ≥2023 after stage 2.ACY remains stable within stages at 2020–2022 and 2023–2024 and persists through post-training, while count-model tense distributions closely match the LMs with cross-entropy of at most 0.36 and 0.78.
  • Internal mechanism: A one-dimensional year–tense subspace appears in residual streams from layers 4 to 22 above the year’s last token, supporting a factual-recall-like mechanism.Interchanging representations for temporally unrelated four-digit numbers yields 90%+ early-layer intervention accuracy, suggesting generalization to year-like numbers beyond temporal contexts.

4 The declarative current year

The declarative current year (DCY) is directly elicited through prompts and aligns with post-training data rather than pre-training cutoffs. Causal analyses find localized mechanisms that differ from the associative task, with non-time-related tokens sometimes exerting the strongest influence.

  • Measurement: The DCY is the average greedy-decoded year across 10 declarative prompts, using continuation prompts for base LMs and instruction prompts for post-trained LMs.The study also measures DCY standard deviation, confidence, and alignment with training-data year distributions.
  • Development: During pre-training, the DCY does not align with the 2023 data cutoff, whereas after SFT it aligns with the post-training cutoff and the most frequent post-training year.The pre-training DCY is also not consistently aligned with the most frequent pre-training years, 2008 and 2010.
  • Modified SFT: 2021 is the no-years SFT LM’s DCY with zero variance, compared with a ±15.0 prompt spread for base OLMo2-1B Stage 2.Removing years from SFT data therefore collapses prompt-to-prompt DCY variance.
  • Causal mechanisms: Causal interventions trace which tokens and layers contribute to the declarative year by applying attention and MLP knockouts and measuring changes in the predicted year prefix.The analysis defines a candidate set of three-digit year-prefix tokens and reports year-shift magnitude as the absolute pre/post-intervention difference.
  • Causal mechanisms: First-position knockouts are often most influential even when the token is not time-related, contrasting with the associative task’s specific influence from year tokens.Low within-group YSM variance indicates that localization is fairly consistent across prompts.

5 Updating the current year in LMs

The section evaluates fine-tuning, in-context year specification, and low-rank weight editing for updating declarative and associative current-year representations. Declarative predictions shift readily, whereas associative predictions largely resist prompting and fine-tuning and require weight editing, which does not simultaneously update both notions.

  • Methods: Three methods target the two current-year notions: year-shifted fine-tuning, specifying the year in context, and low-rank weight editing.ROME updates declarative current-year facts, while MEMIT updates associative year–tense associations.
  • Declarative current year: All three methods accurately shift declarative current-year predictions to target years, with system-prompt specification requiring only a prompt change.Even the control model predicts the correct declarative year in 50.7% of cases and is off by 6.82 years on average when prompted with a target year.
  • Associative current year: 1.4% and 1.7% of 351 prompted instances yield the correct associative current year for the control and system-prompt SFT models, respectively.Across prompted instances, predicted associative years remain within 2019–2030, indicating that prompting alone does not meaningfully move them.
  • Associative current year: Year-shifted fine-tuning shifts the associative current year only over a limited range, after which predictions collapse back near 2020.For target years 2017, 2029, and 2041, associative-current-year errors are within three years; beyond this range, the predictions return near 2020.
  • Associative current year: Weight editing shifts the associative current year within six years of the target for all eight tested years, but the same edit does not update the declarative current year.This makes weight editing effective for the associative task without providing a simultaneous update of both notions.

6 Conclusion · A Associative behavior on more LMs, prompts and output verbs · B Alternative ACY setup with tense before year

The paper shows that associative and declarative notions of the current year are not consistently encoded: the associative notion is structurally ingrained and resistant to post-hoc changes, whereas the declarative notion is shallow and readily shifted. Additional experiments show that associative behavior generalizes across models, prompts, verbs, and tense–year orderings.

  • 6 Conclusion: The declarative and associative tasks probe distinct notions of the current year, revealing that an apparently single concept is not consistently encoded.The associative current year (ACY) links year to verb tense, while the declarative current year (DCY) directly represents the queried year.
  • 6 Conclusion: The ACY is learned during pre-training, accurately proxies the pre-training cutoff year, and resists prompting or fine-tuning.Its structured mechanism links year to verb tense.
  • 6 Conclusion: The DCY is learned during post-training, lacks a consistent internal mechanism, and can be easily shifted via prompting or fine-tuning.ACY-specific weight editing works for ACY but does not generalize to DCY.
  • 6 Conclusion: Because ACY and DCY are decoupled, no existing method succeeds in updating them simultaneously.The contrast reflects greater resistance to modifying concepts ingrained in language structure than surface-level representations.
  • A Associative behavior on more LMs, prompts and output verbs: Across all six tested LMs, the associative task coherently shifts from past to present+future.This extends the pattern observed earlier to a broader set of models.
  • A Associative behavior on more LMs, prompts and output verbs: The associative task remains robust across exact prompts and output verbs, again showing a coherent shift from past to present+future.The expanded evaluation uses a comprehensive list of past and present+future verb tenses rather than a single output-verb set.
  • B Alternative ACY setup with tense before year: When tense appears before the disambiguating year, present and past log-probabilities still rise toward the training cutoff and drop sharply at it for all tested LMs.Thus, the co-occurrence hypothesis remains robust to the order of year and tense.

C Implicit contextual association via factual recall

An implicit associative task tested whether language models recover the current year by recalling the dates of named events rather than reading the year directly. Unlike the main associative task, models showed no temporal tense shift across event years, and factual recall cannot evaluate far-future year updates.

  • Motivation: The implicit task was designed to test whether factual recall of named event dates could reproduce the main associative task’s temporal shift.The year was conveyed implicitly through factual recall rather than stated directly in the prompt.
  • Task setup: The task used 29 event-linked phrases spanning 2010–2028, with 2–3 prompts per year, and inferred time from the next-token verb tense.Events included past and future examples; was/were counted as past, while is/are/will counted as present+future.
  • Results: All four LMs showed no clear increase in present+future tense probability or decrease in past tense probability across event years.Past tense dominated even for future events, while SFT models were more strongly past-dominated than base models.
  • Limitation: Factual evaluation cannot test shifts toward far-future years because prompts about events that have not occurred cannot be constructed.The main ACY task avoids this limitation because verb tense carries temporal information without requiring future-specific facts.

D Factual recency as an implicit current-year probe

The factual recency probe infers the current year from the named answer to recurring-event questions, rather than from verb tense. It finds no consistent implied year across facts: base models usually give older answers, while SFT models mostly refuse.

  • Setup: The probe asks for the most recent instance of 18 recurring facts and converts the named answer into an implied current year.Each fact uses three phrasings, with continuation prompts for base LMs and matching question prompts for SFT LMs, producing 108 prompt templates.
  • Results: No model produces a consistent implied year across the 18 facts, contrary to what coherent current-year encoding would predict.The probe tests whether factual knowledge yields a consistent current-year estimate and how that estimate relates to the training-data cutoff.
  • Results: 4% of prompts elicited the current answer from base LMs, while 81% elicited a real but older answer and 15% produced hallucinations.Base LMs never refused these prompts.
  • Results: 54% of prompts elicited refusals from SFT LMs, while hallucinations were rare at 5%.The supplied results passage states that the implied years of answered SFT prompts continue the contrast with base-model behavior, but its sentence is truncated.

E Obtaining perplexities on the WIKISPAN dataset

The section obtains effective knowledge-cutoff estimates from perplexity over time on WIKISPAN. It combines extracted predictions for eight models with replicated measurements for five models not covered by prior analysis.

  • Method: Effective cutoff predictions for the first eight models were extracted by identifying the month and year with the lowest relative perplexity.This follows Cheng et al. (2024)’s method of measuring perplexity across temporally ordered data versions.
  • Method: The remaining five models were evaluated by replicating WIKISPAN construction and perplexity measurement using the paper’s publicly available code.These models were not covered in the original analysis.
  • Results: The resulting plots for the five replicated models are shown in Figure 10 and match the format of the original Figure 4.Figure 10 reports relative perplexities of language models by month using WIKISPAN.

F N-gram and co-occurrence models

The section constructs N-gram and co-occurrence models from OLMo2 pre-training data to test whether associative current-year behavior reflects corpus statistics. OLMo2 language models show lower cross entropy with these count-based models than with random baselines, suggesting pre-training data shapes the associative current year.

  • Model construction: The models use training-data counts from olmo-mix-1124 and dolmino-mix-1124, corresponding to stages 1 and 2 of OLMo2-1B and OLMo2-7B pre-training.These count-based models are used to compare associative year-tense behavior across training stages.
  • Co-occurrence model: The co-occurrence model splits training sequences into sentences and further splits sentences longer than 20 words to avoid inaccurate long-range word associations.Sentence splitting uses the nltk python library; long unpunctuated sections could otherwise inflate co-occurrence counts.
  • N-gram model: The N-gram model counts tense markers after “In [year] there,” combines was/were as past tense and is/are/will as present+future tense, then applies Laplacian smoothing with an alpha of 1.0.The matching procedure uses a case-insensitive regex with word-boundary requirements.
  • Cross-entropy analysis: OLMo2 LMs have lower cross entropy with corpus-count models than with a random baseline, suggesting the associative current year is shaped by the pre-training data distribution.The comparisons cover N-gram and co-occurrence models constructed from pre-training data at Stage 1 and Stage 2.

G Controls and robustness for the associative year-type subspace … J Training data counts of years

Controls show that the associative year-type subspace is learned, localized, and selectively generalizes, while causal tracing identifies distinct contributions from year and final-token computations. Additional sections document declarative-year prompts and training-data year-counting procedures.

  • G Controls and robustness for the associative year-type subspace: 14% IIA is obtained by randomly interchanging a one-dimensional subspace, matching evaluation pairs whose target tense does not change.This indicates arbitrary subspace interchanges do not systematically control the year-tense mechanism.
  • G Controls and robustness for the associative year-type subspace: 0% IIA across layers in randomly initialized OLMo2-7B indicates high-IIA solutions depend on learned representations rather than DAS alone.The model was evaluated with the same procedure and evaluation set as the year-type experiments.
  • G Controls and robustness for the associative year-type subspace: > 90% IIA in five layers shows that the learned subspace can generalize to four-digit numbers in non-temporal arithmetic contexts.These interchanges succeed in early layers, using year-like numbers that are not used as years.
  • H Causal tracing through attention and MLP knockouts: The largest tense shifts occur when knocking out earlier-layer computations on year tokens and later-layer computations on the final token.Causal tracing across 70 correctly behaving prompts corroborates the year-type localization results.
  • H Causal tracing through attention and MLP knockouts: Future-year prompts show larger knockout effects on the second year token, whereas past-year prompts are less intensely affected there.The passage describes asymmetric responses across past-year and future-year prompts.
  • I Declarative prompts and behavior: Declarative current-year behavior is evaluated with continuation prompts for base models and instruction prompts for fine-tuned models.The respective prompt sets are provided in Table 12.
  • I Declarative prompts and behavior: Ten prompts were selected so both base and SFT OLMo2-1B and OLMo2-7B models responded with a year.The passage highlights the effort devoted to selecting this shared prompt set.

K SFT training OLMo2-1B · L Declarative causal tracing

The paper tests how SFT data transformations affect OLMo2-1B’s declarative current-year behavior and analyzes causal pathways for declarative prompts. SFT experiments remove or shift years in training data, while causal tracing finds no consistent layer or token position for declarative year responses.

  • K SFT training OLMo2-1B: SFT uses the Tulu3 dataset, which OLMo2-1B uses for its SFT training.The experiment tests how training-data content affects the declarative current year.
  • K SFT training OLMo2-1B: The SFT setup uses an effective batch size of 192, learning rate 5e-5 with linear decay and no warmup, 1 epoch, and maximum sequence length 1024.The effective batch size combines per-device batch size 4, gradient accumulation 8, and 6 GPUs.
  • K SFT training OLMo2-1B: The No-years SFT model removes all training examples containing years, defined as all numbers ≥1000, while preserving 76% of training examples.This isolates the effect of year-containing examples in the SFT data.
  • L Declarative causal tracing: Causal tracing is applied to oneshot declarative prompts, with mean magnitude and variance shifts reported in Figure 15.Fewshot declarative prompts are separately provided for current-year and today-date mechanistic analyses.
  • L Declarative causal tracing: Across four declarative prompt types in question and continuation formats, no consistent layer or token position emerges, indicating no stable causal pathway for the declarative current year.The prompt types differ in few-shot examples whose token lengths prevent them from being shown in the figure.
  • L Declarative causal tracing: Declarative current-year prompts use few-shot question and continuation formats with varying month and day, whereas today-date prompts fix the date to June 14 and vary historical examples.These prompt designs are used for mechanistic analysis in Tables 13 and 14.

M Training dynamics on OLMo2-7B · N Prompting fails to shift the ACY across model families and scales

Across OLMo2-7B training, the associative current year remains stable through pre-training while the declarative current year shifts during post-training. Across nine additional Gemma-3 and Qwen-3.5 models, prompting shifts the declarative year effectively in some cases, but associative-year shifting remains difficult and varies non-monotonically with model size.

  • M Training dynamics on OLMo2-7B: The ACY remains stable throughout OLMo2-7B pre-training, while the DCY shifts only during post-training.This pattern matches the earlier OLMo2-1B training dynamics.
  • M Training dynamics on OLMo2-7B: OLMo2-7B training dynamics were evaluated across base pre-training, SFT, RLVR, DPO, and the final Instruct checkpoint.The analysis tracks both ACY and DCY across these training stages.
  • N Prompting fails to shift the ACY across model families and scales: Prompting was extended beyond OLMo2-1B to nine additional LMs across the Gemma-3 and Qwen-3.5 families.Prompting was used because several models were too large to fine-tune or weight-edit with the available compute.
  • N Prompting fails to shift the ACY across model families and scales: The evaluation specified target current years from 1900–2250 and measured exact matches across 351 target years for both ACY and DCY.Each model received the target year through a system prompt.
  • N Prompting fails to shift the ACY across model families and scales: DCY shifting accuracy varies non-monotonically with model size, both across and within model families.This contradicts the hypothesis that larger, more capable models would shift both years more accurately.
  • N Prompting fails to shift the ACY across model families and scales: 100% DCY shifting accuracy was achieved by Qwen3.5-4B, Qwen3.5-9B, and Qwen3.5-27B with system prompting, while ACY shifting remained difficult.High declarative-year performance therefore did not imply comparable associative-year shifting.
Loading 2608.15507v1…