PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Fact Conflicts in LLMs: Reproducibility Insights from Unifying DYNAMICQA and MULAN

Ritajit Dey, Iadh Ounis, Graham McDonald, Yashar Moshfeghi

arXiv:2603.15892v1cs.IRcs.CL

TL;DR

Temporal-fact conflicts prompt divergent conclusions in DYNAMICQA and MULAN about whether external context updates LLM knowledge. This paper reproduces both studies, cross-tests their evaluation frameworks on the other dataset, and evaluates models of different sizes; the findings depend on benchmark setting, with model-size variation also affecting updateability results.

  • Problem

    DYNAMICQA and MULAN report opposite conclusions about temporal-fact updateability using different datasets and measures, leaving their disagreement unresolved.

    Source: §1 · p. 2

  • Method

    The study reproduces both benchmarks, applies each evaluation framework to the other dataset, and extends experiments across Gemma 3 model sizes.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    Both original studies’ conclusions are reproduced, but their conflicting claims do not generalise beyond their original settings; MULAN’s update-success pattern also varies by model size.

    Source: §5 · p. 11, §6 · p. 13

  • Takeaways & Limitations

    Dataset construction, evaluation methodology, and model size shape conclusions about temporal-fact updateability in these benchmarks.

    Source: §5 · pp. 11–13, §6 · p. 13

    DYNAMICQA uses the same relations for static and temporal facts and estimates temporality from edit counts, which can misclassify stable facts.

    Source: §4 · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) often struggle with temporal fact conflicts due to outdated or evolving information in their training data. Two recent studies with accompanying datasets report opposite conclusions on whether external context can effectively resolve such conflicts. DYNAMICQA evaluates how effective external context is in shifting the model's output distribution, finding that temporal facts are more resistant to change. In contrast, MULAN examines how often external context changes memorised facts, concluding that temporal facts are easier to update. In this reproducibility paper, we first reproduce experiments from both benchmarks. We then reproduce the experiments of each study on the dataset of the other to investigate the source of their disagreement. To enable direct comparison of findings, we standardise both datasets to align with the evaluation settings of each study. Importantly, using an LLM, we synthetically generate realistic natural language contexts to replace MULAN's programmatically constructed statements when reproducing the findings of DYNAMICQA. Our analysis reveals strong dataset dependence: MULAN's findings generalise under both methodological frameworks, whereas applying MULAN's evaluation to DYNAMICQA yields mixed outcomes. Finally, while the original studies only considered 7B LLMs, we reproduce these experiments across LLMs of varying sizes, revealing how model size influences the encoding and updating of temporal facts. Our results highlight how dataset design, evaluation metrics, and model size shape LLM behaviour in the presence of temporal knowledge conflicts.

1 Introduction

Temporal facts create conflicts in LLMs’ stored knowledge, but DYNAMICQA and MULAN reach opposite conclusions about their updateability. This study reproduces both benchmarks, tests each framework on the other dataset, and extends the analysis across model sizes.

1 Introduction

  • Temporal facts can leave LLMs with contradictory stored knowledge as information changes over time.The introduction identifies outdated or evolving pretraining information as a source of these conflicts.Source: §1 · p. 1
  • DYNAMICQA reports that dynamic facts resist external context more than static facts, while MULAN finds temporal facts easier to update, using different metrics and context formats.DYNAMICQA measures shifts in output distributions with natural-style passages; MULAN measures changed memorised answers with programmatically constructed statements.Source: §1 · p. 2
  • The study applies each benchmark’s evaluation framework to the other dataset, generating realistic natural-language contexts for DYNAMICQA’s evaluation of MULAN.It also converts question–answer and fact-completion formats to enable comparison across frameworks.Source: §1 · p. 2
  • The study reproduces both benchmarks and examines how dataset design and evaluation methodology shape conclusions about temporal-fact updateability.It also analyzes how model size affects the retention and updating of temporal facts.Source: §1 · pp. 2–3

2 Related Work

Prior work studies factual updates through retrieval, parameter editing, and in-context prompts, and identifies factors that shape receptiveness. DYNAMICQA and MULAN differ in datasets and evaluation, motivating this study’s cross-benchmark comparison.

2 Related Work

  • Knowledge updates use retrieval-augmented generation or parametric editing, while in-context editing can yield more reliable updates than direct parameter edits.MULAN uses in-context editing to study temporal facts.Source: §2 · p. 3
  • Update receptiveness varies with context alignment and logical coherence, and stronger memory for a fact is associated with lower receptiveness to external context.Prior studies also report that models often ignore context contradicting stored knowledge.Source: §2 · p. 3
  • DYNAMICQA uses Wikipedia edit histories, natural passages, and Coherent Persuasion scores, whereas MULAN uses Wikidata facts, constructed statements, and update-success rates.DYNAMICQA reports dynamic facts as more resistant to updating; MULAN reports temporal facts as easier to update.Source: §2 · p. 3
  • The benchmarks operationalize temporality and context receptiveness differently, and this study aligns their datasets and tests each framework on the other while extending evaluation beyond 7B models.MULAN uses time-contingent structured relations; DYNAMICQA derives temporality from Wikipedia edit histories.Source: §2 · p. 4

3 Methodology

The study first reproduces the original evaluation methods, then extends them to Gemma models and applies each framework to the other benchmark. Dataset formats and contexts are adapted to make those cross-benchmark evaluations possible.

3 Methodology

  • The reproduction uses DYNAMICQA’s Coherent Persuasion and Semantic Entropy measures alongside MULAN’s Update Success Rate.It also reports performance across fact types and first-token-probability confidence estimates for MULAN.Source: §3 · pp. 4–5
  • The authors identify a discrepancy between the paper’s and released code’s formulas for group-level output probabilities and follow the code’s approach.The code averages probabilities of answers within each cluster.Source: §3 · pp. 4–5
  • Update Success Rate is the proportion of cases where context changes a memorised fact, with success requiring an exact match to the replacement object.Memorised facts are selected from correctly answered, high-confidence examples; contexts replace the true object with a random alternative from the same relation.Source: §3 · p. 5
  • Semantic Entropy estimates uncertainty by sampling diverse generations, grouping semantically equivalent answers, and computing entropy over those groups.The study uses the authors’ released code without modification for this measure.Source: §3 · p. 5
  • Cross-size reproductions extend both benchmarks to Gemma 3 models with 1B, 4B, and 12B parameters, beyond the original studies’ 7B models.The released benchmark code is extended for Gemma prompts and output processing.Source: §3 · p. 5
  • For cross-benchmark tests, the study converts question and completion templates, excludes disputed DYNAMICQA facts for MULAN-style evaluation, and creates LLM-generated contexts for MULAN.Generated passages are filtered for entailment scores above 0.95, manually checked in a random sample of 200, and paired with semantically similar substitute entities; 2,500 instances per class are sampled from MULAN.Source: §3 · pp. 5–6

4 Experimental Setup

The setup aligns the DYNAMICQA and MULAN datasets and evaluation measures to examine temporal-fact conflicts. It also extends the original 7B-model evaluations to Gemma models of several sizes.

Experimental Setup

  • The experiments address reproducibility, model-size effects on retaining and updating temporal facts, and how dataset and evaluation choices relate to the studies’ conflicting conclusions.The setup section introduces the datasets, evaluation metrics, models, and decoding hyperparameters.Source: §4 · p. 6

Datasets

  • DYNAMICQA uses trivia QA from Wikipedia across 16 relations, while MULAN uses fact completion from Wikidata across 35 relations and five templates per relation.DYNAMICQA groups facts as static, temporal, or disputed; MULAN uses Immutable-1, Immutable-N, and Mutable classes.Source: Table 1 · p. 6, §4 · p. 7
  • For cross-dataset consistency, Immutable-1 is treated as DYNAMICQA’s static class, and Mutable as its temporal class.Immutable-N remains a separate MULAN control class.Source: §4 · p. 7

Evaluation Metrics

  • Evaluation reproduces DYNAMICQA’s accuracy, Coherent Persuasion, Semantic Entropy, and stubborn/persuaded measures, alongside MULAN’s Update Success Rate.Stubborn cases retain the same answer with and without context; persuaded cases change to reflect the context.Source: §4 · pp. 7–8

Models and Hyperparameters

  • The model set includes the original studies’ 7B models and Gemma 3 models at 1B, 4B, and 12B.DYNAMICQA accuracy uses stochastic sampling at temperature 1.0; its Semantic Entropy and Coherent Persuasion measures use 10 samples at temperature 0.5, while MULAN-style evaluation uses greedy decoding.Source: §4 · p. 8

5 Results

The reproduced results confirm the benchmarks’ original opposing conclusions, but cross-dataset tests show that updateability findings do not transfer uniformly. Across Gemma sizes, the updateability pattern also varies with model size, while dataset construction and evaluation choices help explain the divergence.

RQ1: Reproduction

  • On MULAN, mutable facts have lower average F1 and confidence but higher Update Success than immutable-1 and immutable-N facts.This reproduces MULAN’s conclusion that mutable facts are easier to update.Source: Table 6 · p. 10, Table 7 · p. 10, §5 · p. 10
  • DYNAMICQA’s original findings reproduce: dynamic facts have lower accuracy and Coherent Persuasion, fewer persuaded instances, and more stubborn instances than static facts.The reproduction also finds no trend of higher Semantic Entropy for dynamic facts.Source: §5 · p. 10

RQ2: Model size

  • Across Gemma 1B, 4B, and 12B, MULAN Update Success shows no consistent pattern: mutable facts are less successfully updated at 1B and 12B, but not at 4B.The intermediate-size model shows the opposite behavior, making the pattern non-monotonic within this family.Source: Table 9 · p. 11, §5 · p. 11

RQ3: Cross-dataset evaluation

  • MULAN’s mutable-fact findings generalize under DYNAMICQA’s evaluation, but DYNAMICQA’s temporal-update pattern transfers only partially under MULAN’s evaluation.The cross-benchmark results vary across fact types and models, rather than establishing a single direction for temporal updateability.Source: Table 10 · p. 12, Table 11 · p. 12, §5 · p. 12

RQ3: Dataset and evaluation differences

  • DYNAMICQA’s temporal class includes 1,369 of 2,495 facts with only two edits, which may reflect corrections or synonyms rather than genuine temporal change.The authors also identify template sensitivity and differing operational definitions of temporality as factors that plausibly explain divergent findings.Source: §5 · p. 13

6 Conclusions

The study reproduces the original opposing benchmark findings, but cross-dataset tests and model-size analyses show that updateability conclusions are contingent on experimental settings. Whether external context reliably resolves temporal knowledge conflicts remains open.

Conclusions

  • Both original benchmark claims were confirmed, but their conflicting conclusions do not generalize beyond their original settings.The study identifies dataset construction, definitions of temporality, evaluation metrics, and model size as factors shaping the findings.Source: §6 · p. 13
  • Neither benchmark alone provides a complete picture of temporal knowledge conflicts in LLMs.Whether external context reliably resolves these conflicts remains an open question.Source: §6 · p. 13
  • The study’s cross-benchmark and model-size analyses find that which fact class is easier to update can vary with experimental setting.The authors conclude that benchmark-specific conclusions are highly contingent on those settings.Source: §6 · p. 13
Loading 2603.15892v1…