PaperlayerView exact revision on arXiv

Source-linked AI summary

Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense

arXiv:2609.36931v1cs.CL

TL;DR

Prior work links LLM-output non-determinism to factors such as hardware and batching, while the hidden, time-varying date in system prompts is an overlooked potential source. The study swept system-prompt dates from January 1 to December 31, 2024, changing only the date while keeping identical prompts and the rest of the configuration fixed and deterministic. Across 9 models and four tasks, dates changed performance and rankings, with up to 14% accuracy variation on math reasoning; CoT and few-shot prompting did not reduce sensitivity.

  • Problem

    Prior work links LLM-output non-determinism to factors such as hardware and batching, while the hidden, time-varying date in system prompts is an overlooked potential source.

    Source: §1 · p. 1

  • Method

    The study swept system-prompt dates from January 1 to December 31, 2024, changing only the date while keeping identical prompts and the rest of the configuration fixed and deterministic.

    Source: §3 · p. 2

  • Results

    Across 9 models and four tasks, dates changed performance and rankings, with up to 14% accuracy variation on math reasoning; CoT and few-shot prompting did not reduce sensitivity.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The authors recommend removing the date from the chat template when possible, or fixing it and reporting it with results.

    Source: §6 · p. 5

    Evidence covers four representative tasks and selected models; proprietary evaluation tests only GPT-5.1 over one week, and other date formats, positions, or years remain untested.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies solely with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques -- chain-of-thought and few-shot prompting -- do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.

1 Introduction

Prior sources of LLM non-determinism hinder reproducible benchmarking. This paper examines whether hidden, time-varying dates in system prompts change evaluation results and finds substantial performance variation and ranking shifts.

1 Introduction

  • Prior work associates LLM-output non-determinism with batch size, numerical precision, and hardware differences.These factors hinder reproducibility in benchmarking.Source: §1 · p. 1
  • Date-only changes to hidden system-prompt metadata alter evaluation results, revealing an overlooked source of LLM non-determinism.The paper asks whether evaluations fluctuate across days when other settings remain identical.Source: §1 · p. 1
  • Across 9 recent LLMs and 6 benchmarks, date changes shift results by up to 6% on MCQA, with larger reported effects on generation tasks.The introduction reports up to 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation.Source: §1 · p. 1
  • The authors call for evaluation protocols that account for hidden prompt metadata to support reproducibility and fair comparisons.Figure 1 illustrates performance changes when only the current date changes.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

Prior work documents evaluation instability from decoding, prompt-formatting, and system-level configuration choices. This paper distinguishes hidden, time-varying date insertion as another source of variance.

2 Related Work

3 Experimental Setup

The experiments isolate the current date by varying it across 2024 while holding prompts and other settings fixed. The evaluation spans nine recent models, six datasets, four task types, and task-appropriate metrics.

3 Experimental Setup

  • The date is swept across every day of 2024 while identical prompts and deterministic settings are otherwise held fixed.The year is selected as a representative period of rapid LLM development and benchmarking.Source: §3 · p. 2
  • The suite includes MMLU, GPQA, ARC-Challenge, GSM8K, HumanEval, and WMT English-to-German, Finnish, and Czech translation.MCQA questions are checked to be time-independent; tasks grade single answers, generated programs, or full translations.Source: §3 · p. 2
  • Nine recent models from varied families, sizes, and capabilities are evaluated.The lineup includes Llama 3.1 Instruct, Gemma 3 Instruct, Qwen3, Qwen3-Next, Phi-4, and GPT-OSS.Source: §3 · p. 2
  • Evaluation reports MCQA accuracy and ECE, GSM8K accuracy, HumanEval pass@1, and MT BLEU and chrF.ECE uses 10 equal-width bins.Source: §3 · p. 2

4 Results

Date changes shift results across MCQA, math reasoning, execution-graded code, and full-output translation, with variation also changing model rankings. The reported effects persist when all other settings are fixed.

4 Results

  • Date-driven performance variation appears in MCQA, math reasoning, execution-graded code, and full-output translation, so the effect is not limited to MCQA.The translation and code results extend the pattern to full-output scoring and unit-test evaluation.Source: §4 · pp. 2–3

The Effect Is Larger for Reasoning Tasks

  • GSM8K deltas average 7.75%, versus 2.52% on MCQA; longer generations give the date repeated opportunities to bias intermediate tokens and cascade into different answers.The paper notes that these deltas make accuracy reported on different days difficult to compare.Source: §4 · pp. 2–3

MCQA Results Vary with Date Changes

  • Up to 6% accuracy differences occur across dates on MCQA, where accuracy and ECE also vary and model rankings reorder.Figure 2 summarizes MMLU accuracy and ECE across 2024 dates for the top five models.Source: §4 · p. 2, Table 1 · p. 3, Figure 2 · p. 3

Execution-Graded Code Still Shifts

  • On HumanEval, pass@1 changes by 4.81% on average and by up to 7.32% for the most affected models, despite execution-based grading.The metric is based on running generated code against unit tests.Source: §4 · p. 3

The Effect Holds for Full-Output Metrics

  • For machine translation, average date deltas reach 1.88 BLEU and 1.33 chrF, showing shifts on full-output metrics.The authors state that the deterministic setup leaves the hidden date as the changing factor.Source: §4 · p. 3

5 Analysis

Date sensitivity has no consistently favorable date, reaches proprietary models, and persists under few-shot prompting; chain-of-thought amplifies it. Against other tested sources of variation, date effects exceed batch size, precision, and hardware, while matching option order and prompt-wording changes.

5.1 No Date Is Consistently Better

  • Date and accuracy correlations are near zero over time, across datasets, and across models; dates move paired accuracies in the same direction on only 51% of dates.The median date–accuracy correlation is 0.02, and 50% same-direction movement is chance, so no date is consistently better.Source: §5.1 · pp. 3–4, Table 3 · p. 4

5.2 The Effect Reaches Proprietary Models

  • GPT-5.1 varies by up to 4% on GPQA across a week, validating date sensitivity on a proprietary model.Repeated runs at a fixed date produced identical scores in all 42 runs.Source: §5.2 · p. 4, §5.3 · p. 4, Figure 3 · p. 4

5.3 Chain-of-Thought Amplifies Date Sensitivity

  • Chain-of-thought amplifies date sensitivity: date-driven changes to initial tokens cascade into different reasoning paths and final answers.The authors attribute the cascade to autoregressive generation, which gives the date more surface to act as a confounder.Source: §5.3 · p. 4

5.4 Few-Shot Learning Does Not Help Either

  • With 5-shot prompting, average accuracy deltas remain 2.27%, versus 2.52% zero-shot, so few-shot prompting does not remove date gaps.The gaps persist across all models.Source: §5.4 · pp. 4–5, Table 4 · p. 5

5.5 Comparison Against Other Sources of Non-Determinism

  • Batch size, numerical precision, and hardware introduce less variability than date changes, while option-order variability is comparable.The comparison used Llama 3.1 (8B) on MMLU and measured accuracy and ECE variation with coefficient of variation.Source: §5.5 · p. 5, Table 5 · p. 5

5.6 Other Variations in the System Prompt

  • Changing system-prompt wording produces the same 0.78% accuracy CV as date changes, supporting sensitivity to system-prompt modifications generally.The date change occurs without the user’s knowledge, unlike intentional prompt engineering.Source: §5.6 · p. 5

6 Conclusion

The paper identifies hidden current-date injection into system prompts as a source of LLM evaluation variability across models and tasks. Date changes alter performance and rankings, and the authors recommend accounting for this metadata in evaluation protocols.

6 Conclusion

  • Across 9 models, 6 datasets, and four tasks, hidden date changes alter performance and reshuffle leaderboard rankings.Reported worst-to-best effects reach 6% accuracy on MCQA, 14% accuracy on math reasoning, 7% pass@1 on code generation, and 2.84 BLEU on machine translation.Source: §6 · p. 5
  • Date effects surpass variability from batch size and numerical precision, while neither chain-of-thought nor few-shot prompting reduces sensitivity.Chain-of-thought amplifies the effect.Source: §6 · p. 5
  • When possible, remove the date from the chat template without changing the rest; otherwise, fix the date and report it with results.Source: §6 · p. 5

Limitations

The study’s empirical scope is limited to selected models and datasets, with proprietary evaluation covering GPT-5.1 over one week. It also uses “non-determinism” for expected output variability, while noting that standard LLMs are deterministic when their full configuration is identical.

Limitations

  • The study covers selected models and datasets; proprietary evaluation is limited to GPT-5.1 over one week, so broader generalization remains open.The experiments cover four representative tasks; open-ended tasks such as dialogue and summarization remain to be studied.Source: Limitations · p. 6
  • Date injection was tested only in one fixed format and prompt position within 2024; sensitivity to other formats, positions, or years remains untested.Source: Limitations · p. 6

A On the Use of the Term “Non-Determinism”

  • The paper uses “non-determinism” for expected output variability, while standard LLMs are technically deterministic when the full configuration stays identical.The authors suggest “non-reproducibility” or “practical nondeterminism” may better describe real-world evaluation challenges.Source: A On the Use of the Term… · p. 8

B On the Hidden Injection of the Current Date

The paper examines dates that API providers or chat templates add invisibly above user-defined system prompts. Because this date changes over time while other settings can remain fixed, the study investigates whether it alters benchmark evaluations.

B On the Hidden Injection of the Current Date

  • The paper calls the date injection “hidden” because the date is added above the user-defined system message and is not visible to users or researchers.Providers prepend dates for temporal context, and open-weight chat templates can insert them above the user prompt as well.Source: B On the Hidden Injection of the… · p. 8
  • Qwen3 can receive a date through server-side provider injection even though its default chat template does not include one.When called through OpenRouter with an empty system prompt, no tools, and no internet access, Qwen3 reported the current date.Source: B On the Hidden Injection of the… · p. 8, Figure 5 · p. 8
  • With the system prompt and configuration fixed, the injected date is the only prompt detail that changes over time, motivating a study of its evaluation effects.The paper asks whether benchmark results fluctuate across dates under otherwise identical settings.Source: B On the Hidden Injection of the… · p. 8

C Experimental Details

The experimental details define the calibration measure, control decoding variability, describe task-specific prompt templates, and test sensitivity to system-prompt wording.

C.1 Expected Calibration Error

  • ECE measures alignment between confidence and accuracy using 10 equal-width confidence bins.The bins group instances; the calculation compares predicted and true labels with each instance’s confidence.Source: §C.1 · p. 9

C.2 Decoding

  • At a fixed date, repeated generations are identical; the reported variation therefore comes only from changing the date.MCQA uses next-token probabilities without sampling, while open-ended tasks use default temperature and top-p, fixed seeds, and deterministic CUDA operations.Source: §C.2 · p. 9

C.3 Prompts

  • Task prompts use model-specific chat tokens and parse option labels, final numbers, or tested code outputs, with translation prompts set by target language.The templates cover MCQA, chain-of-thought MCQA, GSM8K, HumanEval, and machine translation.Source: §C.3 · p. 9, Figure 6 · p. 9, Figure 7 · p. 9

C.4 System-Prompt Wordings

  • The system-instruction test varies wording across six versions while keeping all other settings fixed.The versions range from an empty prompt to instructions specifying assistant helpfulness or multiple-choice expertise.Source: §C.4 · p. 10

D Ensuring Questions Are Not Time-Dependent

To check that date sensitivity is not driven by time-dependent questions, the authors use GPT-OSS (120B) to assess every question in the evaluated datasets.

D Ensuring Questions Are Not Time-Dependent

  • GPT-OSS (120B) classified none of the evaluated dataset questions as time-dependent, supporting the conclusion that observed performance variation is not due to question timing.Table 6 reports the number and percentage of questions classified as time-dependent in each dataset.Source: D Ensuring Questions Are Not… · p. 10, Table 6 · p. 10

E Detailed Results

Date changes affect both accuracy and calibration, with inconsistent answers generally less certain, though some models remain highly confident. Omitting the date keeps results within the observed date-specific range, while preliminary results indicate that changing user location can also affect performance.

E Detailed Results

  • Inconsistent MMLU predictions have lower average confidence than consistent predictions, although Gemma 3 (4B) assigns 88.86% confidence to inconsistent predictions.Confidence and calibration alongside accuracy can help flag results prone to shifts under hidden factors such as the date.Source: E Detailed Results · pp. 10–11
  • 6.15% is the maximum ECE difference between the worst and best dates across models and datasets.Table 8 summarizes these date-based ECE differences across models and datasets.Source: Table 8 · p. 11, E Detailed Results · p. 11
  • Without date injection, performance falls within the range observed across dates, so removing the date can remove it as a source of variability.Table 9 reports accuracy and ECE without the injected date.Source: Table 9 · p. 11, E Detailed Results · p. 11

F Other Metadata in the System Prompt

  • Changing user location in the system prompt also produced performance differences comparable in magnitude to the date effect in preliminary Llama 3.1 (8B) MMLU experiments.The accompanying figure examines the impact of different user locations on MMLU performance.Source: F Other Metadata in the System Prompt · p. 11, Figure 14 · p. 15
Loading 2609.36931v1…