PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs

Jonathan Zheng, Zirui Shao, Alan Ritter, Wei Xu

arXiv:2609.00184v1cs.CLcs.LG

TL;DR

Static pretraining leaves LLM knowledge outdated, while existing update evaluations face contamination or conflicts with rigid world knowledge. The paper introduces ParallelEvents and Synapse, and reports that Synapse outperforms strong baselines by over 14.23% while mitigating hallucinations.

  • Problem

    Static pretraining can produce outdated responses, while existing evaluations are vulnerable to contamination or rely on counterfactual edits that conflict with correlated facts and commonsense associations.

  • Method

    ParallelEvents provides verified plausible future worlds for temporal updating, while Synapse uses synthetic data for mid-training and instruction tuning.

  • Results

    Over 14.23%, Synapse outperforms strong baselines across editing settings, including insertion of 1,536 facts and evaluation on MQUAKE-REMASTERED.

  • Takeaways & Limitations

    Synthetic simulation-based training supports scalable knowledge insertion while mitigating hallucinations and improving editing performance across settings.

  • Takeaways & Limitations

    Counterfactual-edit benchmarks can conflict with correlated facts and commonsense associations acquired during pretraining.

Abstract

from arXiv · show

Large language models (LLMs) rely on static pretraining corpora, causing their knowledge to become outdated over time. Existing approaches for evaluating knowledge edits either suffer from rapid contamination or rely on counterfactual edits that conflict with rigid existing knowledge. In this work, we propose a synthetic, simulation-driven framework for studying knowledge insertion in LLMs. We introduce {\sc ParallelEvents}, a benchmark of fictional yet realistic future worlds that generates coherent event trajectories for controlled evaluation, avoiding contamination while preserving consistency. Building on this dataset, we develop {\sc Synapse}, a training framework that uses model-generated data to update model parameters via mid-training and instruction tuning. This synthetic pipeline enables scalable knowledge integration without costly human-curated data. Empirically, {\sc Synapse} outperforms existing methods by 14.23\%, demonstrating that simulation-based synthetic training leads to robust and coherent knowledge insertions.

1 Introduction

The paper addresses temporal knowledge updating with a synthetic benchmark of coherent future worlds and a scalable synthetic-training framework. PARALLELEVENTS avoids contamination and counterfactual inconsistency, while SYNAPSE updates model parameters through mid-training and instruction tuning.

  • Motivation: Static pretraining makes LLM knowledge outdated, while existing update settings cover limited changes and risk contamination or inconsistency.Prompted updates can conflict with parametric knowledge; model editing has scalability and side-effect limitations.
  • PARALLELEVENTS: PARALLELEVENTS generates plausible, unseen future worlds with locally and globally consistent events for controlled temporal evaluation.Its fictional events avoid large-scale propagation into training corpora and support reasoning over evolving states.
  • SYNAPSE: SYNAPSE uses scalable synthetic generation for mid-training and instruction tuning to integrate new facts while mitigating outdated responses, hallucinations, and unwarranted abstentions.The framework also analyzes training regimes and synthetic mixtures to balance knowledge acquisition and behavioral alignment.
  • Benchmark motivation: 38.43% of MQUAKE-CF facts and 10.16% of COUNTERFACT facts appear in at least 1,000 Wikipedia pages, compared with 0.39% of PARALLELEVENTS entities.The co-occurrence statistics characterize overlap with pretraining knowledge and motivate synthetic future-world evaluation.
  • Results: 14.32% improvement over MeLLo on complex events demonstrates SYNAPSE’s reported advantage over a state-of-the-art baseline.The comparison is reported for complex-event evaluation.

2 Preliminaries

This section frames knowledge updating as incorporating new factual triples while maintaining existing knowledge and reviews temporal and counterfactual benchmarks. It highlights that counterfactual edits can conflict with correlated pretrained knowledge and complicate comparisons across model knowledge cutoffs.

  • Benchmark motivation: Evaluation benchmarks should isolate incorporation of new information while preserving existing knowledge independently of model training time.This design addresses the mismatch between static evaluation assumptions and evolving deployment settings.
  • Knowledge Representation and Updating in LLMs: Knowledge updating incorporates a set of new factual triples into an updated model while maintaining existing knowledge.Facts are represented as subject–relation–object triples, and updates may introduce or modify triples.
  • Existing Knowledge Updating Benchmarks: Time-stamped external-knowledge benchmarks capture changes such as political transitions, neologisms, newly named entities, and concept shifts.The passages note that these datasets have a short effect and complicate comparisons between models with different knowledge cutoffs.
  • Existing Knowledge Updating Benchmarks: Counterfactual edits reduce pretraining leakage risk but can conflict with correlated facts and commonsense associations acquired during pretraining.The paper gives the United Kingdom–Europe/Oceania edit as an example of an inconsistent knowledge update.

3 Time-Insensitive Evaluation of LLM Knowledge Insertions

PARALLELEVENTS provides a time-insensitive benchmark of plausible future worlds, representing interconnected events and testing direct recall, multihop composition, and causal reasoning. Its event graphs are generated from structured schemas, verified for consistency, and converted into manually checked questions.

  • Knowledge Representation: The benchmark encodes attributes, entity relations, and directed causal edges to represent event properties, connections, and ripple effects.Causal edges explicitly link main events through cause-and-effect relationships.
  • Data Validation: Structured instances generate new events under known constraints, followed by manual verification of local and global logical coherence.Questions are also manually verified and rewritten for diverse phrasing and answer formats.
  • Benchmark Construction: PARALLELEVENTS simulates plausible future events across sports, natural disasters, elections, economic crises, and city incorporations.The dataset covers scenarios from 2030 to 2035 and includes 41 event-centric knowledge graphs.
  • Benchmark Construction: 41 event-centric knowledge graphs contain 24,530 entities and 15,885 relations spanning 2030–2035 scenarios.Approximately 35% of entities are manually edited to ensure logical consistency.
  • Evaluation Tasks: Evaluation questions test single-hop recall, multihop reasoning over triple chains, and causal reasoning about downstream event consequences.Multi-hop questions average 3.06 hops, while causal questions assess effects that emerge from event dynamics rather than appearing explicitly in the graph.

4 SYNAPSE: Synthetic Augmentation for Preference-Steered Editing

SYNAPSE treats temporal knowledge insertion as synthetic data generation plus preference-steered parameter updating. It combines long-form event text, reasoning questions, preferred responses, and abstention examples to acquire new facts while reducing outdated, hallucinated, or unnecessary responses.

  • Motivation: Existing retrieval and in-context methods achieve 17.09% accuracy on PARALLELEVENTS and over-abstain on 72.53% of recent or future-related queries.These methods can fail when predicates are missing or difficult to traverse, motivating parameter updates for known facts.
  • Synthetic Data Generation: SYNAPSE generates approximately 1000-word synthetic news articles from new event facts for mid-training with next-token prediction.The articles simulate realistic reports of world events and are verified against a unified fact base and knowledge graph.
  • Preference Data: SYNAPSE constructs single-hop, multi-hop, and causal question types from edited fact triples and graph paths.Causal examples use consequence triples, while multi-hop examples sample relational paths with at least two steps.
  • Preference Data: Synthetic preferred responses include chain-of-thought rationales and final answers, while dispreferred responses include outdated values, abstentions, and hallucinations.This contrast supplies the preference signal for updating responses to inserted facts and unknown queries.
  • Training Objective: Direct Preference Optimization reinforces acquired knowledge, discourages incorrect or unnecessary abstentions, and enforces abstention for unknown information.An auxiliary supervised fine-tuning term helps preserve general instruction-following and safety behaviors.

5 Experimental Setup

The experiments vary the number of inserted facts and evaluate question-answering accuracy across causal, multihop, and single-hop tasks. They compare SYNAPSE with parameter-updating, in-context, and graph-traversal baselines on three instruction-tuned models.

  • Evaluation Design: Experiments insert 20, 150, 542, or 1,536 facts covering 1, 15, 29, or 41 events, respectively.Evaluation sets contain 48, 353, 1,314, and 2,658 questions across causal, multi-hop, and single-hop categories.
  • Models and Data: Experiments evaluate LLaMA-3.1-8B Instruct, Olmo-3-7B Instruct, and Gemma-3-4B Instruct using synthetic datasets generated with GPT-4.1.For each event, the study generates 6–15 articles, with total article counts increasing across fact-insertion scenarios.
  • Baselines: Baselines include LoRA, single-layer instruction tuning, in-context methods ICE and IKE, and graph-traversal methods such as PokeMQA, DeepEdit, and MeLLo.The comparison includes both weight-modifying and prompt- or retrieval-based approaches.

6 Results

SYNAPSE achieves the strongest performance on complex temporal knowledge updating, especially causal reasoning, while retrieval augmentation addresses its single-hop numerical-recall weakness.

  • Figure 5 compares token probabilities for the base model, ICL, and SYNAPSE on a causal future-event question.SYNAPSE propagates the correct answer early, around layer 20, and assigns it high probability.
  • 17.09% accuracy and 72.53% abstention: graph-based retrieval and in-context methods perform poorly on complex PARALLELEVENTS updates.These methods often fail to retrieve relevant facts in the benchmark setting.
  • 32.65% causal accuracy: ICL baselines are much weaker on causal questions than on single-hop questions, where they reach 90.86%.
  • 73.67% for LLaMA-3.1, 64.92% for Olmo-3, and 79.07% for Gemma-3: SYNAPSE achieves the highest overall accuracy on relevant questions.
  • SYNAPSE outperforms ICL methods on higher-reasoning questions, with a 53.73% advantage on causal questions.The cited analysis also reports frequent ICL abstention on these questions.
  • Adding 10 retrieved facts closes SYNAPSE’s single-hop gap and yields top-2 performance across all PARALLELEVENTS question types.The hybrid model retains SYNAPSE’s advantage on multi-hop and causal reasoning.

7 Ablation Studies

Ablations show that SYNAPSE remains effective as edit scale increases, generalizes to counterfactual editing, and benefits from TULU-3 to recover general capabilities and abstention.

  • Edit Scale: Only a 1.2% drop from 542 to 1,536 inserted facts: SYNAPSE degradation plateaus while baselines decline by 10.08% on average.SYNAPSE achieves the best performance across all edit-scale settings.
  • General Capabilities: 4.34% average gain: adding TULU-3 to SYNAPSE improves abstention and general knowledge across models and benchmarks.The combined model also exceeds the base LLaMA-3.1-8B-Instruct model on instruction following.
  • Model Generalization: 85.02% train and 75.02% test accuracy: SYNAPSE strongly generalizes to MQUAKE-REMASTERED counterfactual edits.Test accuracy exceeds GWalk by over 41.67%, while Unedited accuracy decreases to 52.07%.

8 Related Work

Prior knowledge-updating methods address temporal drift through retrieval, editing, fine-tuning, and memory mechanisms, but benchmarks largely remain focused on isolated fact corrections.

  • Existing approaches include localized editing, in-context updates, retrieval or graph augmentation, lightweight fine-tuning, instruction learning, context distillation, and episodic memory.
  • Synthetic data has been studied for pretraining, adaptation, instruction following, alignment, reasoning, time-series modeling, and event detection.
  • Prior work had not explored synthetic generation for updating LLMs on future events.

9 Conclusion

The paper introduces PARALLELEVENTS and SYNAPSE for scalable, coherent temporal knowledge updating, while identifying capability trade-offs and responsible-deployment risks.

  • PARALLELEVENTS is a benchmark grounded in plausible future events and designed for reasoning over newly emerging information.
  • SYNAPSE uses synthetic data generation for both pre-training and post-training, outperforming strong baselines by over 14.23%.It remains effective across large-scale insertion settings and MQUAKE-REMASTERED.
  • Synthetic events are manually reviewed to improve plausibility and logical consistency, although the generation process may still introduce biases.
  • Realistic synthetic knowledge carries misuse risks, including convincing misinformation or fabricated facts that could manipulate downstream models.The release labels events as synthetic and uses research-oriented licenses to reduce misuse.
  • Knowledge updating slightly decreases general LLM capabilities, with the largest degradations in out-of-domain math and code topics.The paper characterizes this as a trade-off involving knowledge updating, reasoning capability, and reliability.
  • Responsible deployment requires attention to errors in real-time decisions, synthetic-data biases, fairness, transparency, accountability, and societal impact.

B Limitations

The evaluation is limited by synthetic-data scope, offline and batch-only updating, computational constraints, and incomplete coverage of models and real-world continual learning.

  • Dataset scope and design: PARALLELEVENTS covers six primary event topics, and its deliberately limited automation requires approximately 350 hours of construction.The dataset includes highly complex events despite its narrow topic coverage.
  • Training methodology: SYNAPSE is evaluated only in an offline DPO setting, leaving online updating and reward-model training for hypothetical future events unexplored.The authors suggest periodically refreshing the training set for dynamic updates and identify online methods as future work.
  • Data and compute costs: The approach requires synthetic SFT and preference-data generation, although this overhead is substantially lower than continual pretraining requiring trillions of tokens.For moderate factual insertions, generation and training can be completed in a day.
  • Evaluation scope: Evaluation covers selected insertion sizes and primarily LLaMA-3.1-8B, with ablations restricted to the 150-fact setting because larger experiments are costly.The study also uses a 100-fact MQUAKE setting but does not report broad model coverage.
  • Sequential updates and catastrophic forgetting: The study evaluates batch insertions rather than sequential multi-round updates, so retention of earlier facts and general performance across update rounds remains unmeasured.The authors specifically leave open whether earlier-inserted facts degrade after additional updates.
  • Scale and real-world applicability: Factual performance decreases as inserted facts grow from 10 to 1,536, and fully synthetic evaluation may not capture larger-scale or real-world continual learning.Instruction-tuning preference data helps retain instruction-following ability, but the authors expect degradation to compound for combined insertions.

D Benchmark Details

PARALLELEVENTS is designed to test coherent temporal knowledge insertion through synthetic, verified event worlds and questions spanning exact recall, multi-hop reasoning, and causal consequences.

  • Benchmark motivation: Existing counterfactual benchmarks contain strongly co-occurring entity pairs, while PARALLELEVENTS uses fewer such pairs to reduce conflicts with pretrained associations.One cited counterfactual pair co-occurs in 120,340 Wikipedia articles, whereas synthetic events are designed to avoid similarly rigid associations.
  • Dataset composition: The dataset covers natural disasters, elections, sporting events, economic crises, city incorporations, and creative works across fictional event trajectories.Its event inventory includes 41 events in the largest insertion setting.
  • Verification: Two in-house annotators verify realism, entity attributes, temporal consistency, stereotypes, and bias, while election counts receive human-guided generation.The reported inter-annotator agreement for selecting editable JSON fields is Cohen’s κ = 0.8298.
  • Baselines and protocols: The benchmark compares in-context retrieval, fine-tuning, model editing, and SYNAPSE under specified retrieval and training configurations.Finetuning baselines use matched hyperparameters, while model-editing results are reported only for the 542-fact LLaMA-3.1-8B setting.
  • Headline benchmark result: SYNAPSE with 10 retrieved facts achieves the highest accuracy across total, single-hop, multi-hop, and causal questions after inserting 150 facts.Synthetic questions in its DPO set overlap with evaluation questions in only 1.79% of instances.

H Additional Results

Additional analyses show that SYNAPSE remains strong across insertion scales and model families, especially for causal and multi-hop reasoning, while exposing trade-offs in recall, general knowledge, and baseline design.

  • Model editing: 6.31% total accuracy is achieved by model-editing baselines in the 542-fact setting, alongside a 4.85% abstention rate on unanswerable questions.The methods often fail to incorporate new events and increase hallucinations.
  • Scaling results: 85.55% overall accuracy and 95.00% causal accuracy are achieved by SYNAPSE for 150 inserted facts.The reported gains over baselines are largely driven by causal reasoning accuracy.
  • Scaling results: 78.70% overall accuracy is achieved with SYNAPSE plus 10 retrieved facts for 1,536 inserted facts, including 98.2% single-hop and 83.50% causal accuracy.The same setup reports 46.2% multi-hop accuracy.
  • Hybrid comparisons: SYNAPSE outperforms matched fine-tuning-plus-retrieval hybrids across Llama, OLMo, and Gemma, with the largest gains on multi-hop and causal questions.The comparison gives all methods equal access to retrieved context.
  • Retrieval and reasoning: 64.5% of cases where in-context retrieval performs better are single-hop exact-numeric queries, while SYNAPSE is more consistent across question dynamics.The analysis suggests parameter updates improve reasoning on new facts, whereas retrieval remains stronger for exact lookup.
  • General knowledge: More than 5% performance drops occur in STEM categories, while synthetic historical preference pairs produce a 3.68% gain on historical questions.Technical content is underrepresented in SYNAPSE’s generated-data distribution.

I.6 General Preference Datasets

General preference-data experiments show that preference-set choice and training order affect the balance between event-question performance, abstention, and general knowledge retention.

  • Training order: Training on Event-Preference first and then the general preference set yields the best overall results among the tested training orders.The experiments compare dataset order, step-level alternation, and checkpoint-level alternation on LLaMA-3.1-8B with 150 facts.
  • Preference-dataset comparison: TULU-3 maintains general knowledge understanding more effectively than Helpsteer2 while both general preference datasets improve event question-answering accuracy.The comparison uses MMLU-Pro performance as the indicator of general understanding.
  • Alternation strategy: Checkpoint-level alternation achieves 77.90% total question accuracy and an 82.85% abstention rate.Step-level alternation instead produces abstention accuracy of 10% or less.
  • Training trade-offs: Alternating checkpoints is slower and yields smaller gains in question-answering and MMLU-Pro accuracy, so the paper primarily reports sequential Event-Preference followed by TULU-3 training.The reported choice reflects a trade-off between abstention behavior and training efficiency.
Loading 2609.00184v1…