Source-linked AI summary
Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs
Atahan Özer, Çağatay Yıldız
TL;DR
The paper studies how LLMs answer questions when knowledge evolves and contexts contain outdated or conflicting facts. It introduces two temporal benchmarks and an agentic method that organizes document knowledge into external structured memory. The authors report that this method improves performance across both benchmarks, while ICL and RAG remain vulnerable to temporal noise.
Problem
LLMs have training-data cutoffs, and temporal ICL remains underexplored despite challenges with long inputs, answer position, relevant-context selection, and accumulating facts.
Source: §1 · pp. 1–2Method
The paper introduces Temporal Wiki and Unified Clark and incrementally organizes document facts into structured external knowledge for temporally coherent reasoning without re-training.
Source: §1 · p. 2, §2.2.1 · p. 3, §2.2.2 · p. 4Results
Takeaways & Limitations
The findings support structuring and indexing facts as an approach to reasoning over dynamic information, while both ICL and RAG remain vulnerable to temporal noise.
Source: §4 · p. 8
Abstract
from arXiv · showhide
Large language models (LLMs) exhibit remarkable capabilities in question answering and reasoning thanks to their extensive parametric memory. However, their knowledge is inherently limited by the scope of their pre-training data, while real-world information evolves continuously. Updating this knowledge typically requires costly and brittle re-training, or in-context learning (ICL), which becomes impractical at scale given the volume and volatility of modern information. Motivated by these limitations, we investigate how LLMs perform when exposed to temporal text corpora, or documents that reflect evolving knowledge over time, such as sports biographies where facts like a player's "current team" change year by year. To this end, we introduce two new benchmarks: Temporal Wiki, which captures factual drift across historical Wikipedia snapshots, and Unified Clark, which aggregates timestamped news articles to simulate real-world information accumulation. Our analysis reveals that LLMs often struggle to reconcile conflicting or outdated facts and can be misled when multiple versions of a fact appear in context. To address these issues, we propose a lightweight, agentic framework that incrementally builds a structured, external memory from source documents without requiring re-training. This knowledge organization strategy enables models to retrieve and reason over temporally filtered, relevant information at inference time. Empirically, our method outperforms ICL and RAG baselines across both benchmarks, especially on questions requiring more complex reasoning or integration of conflicting facts.
1 Introduction
The paper examines how LLMs handle evolving knowledge when their parametric memory may be outdated and context contains temporal conflicts. It introduces two benchmarks and a knowledge-organization approach to support reasoning over temporally coherent information.
1 Introduction
- LLM knowledge is limited by training data and cut-off dates, while re-training and continual adaptation become increasingly resource-intensive as models scale.In-context learning is widely used for temporal queries, including through web search.Source: §1 · p. 1
- Temporal ICL remains relatively underexplored and faces longer-input degradation, answer-position sensitivity, and difficulty identifying relevant context.When supporting and conflicting evidence co-occur, models—especially larger ones—tend to fall back on parametric knowledge.Source: §1 · pp. 1–2
- Earlier temporal work studied temporal facts, misalignment, or reasoning, but largely overlooked continually accumulating and integrating new facts.Recent Wikipedia- and news-related approaches also do not study conflicting knowledge or list answers as temporal task requirements.Source: §1 · p. 2
- The paper introduces Temporal Wiki and Unified Clark and reports that its incremental organization of external knowledge yields consistent improvements on tasks adapting to changing facts.The approach organizes document knowledge into structured text and does not require re-training.Source: §1 · p. 2
2 Methodology
The methodology develops benchmarks for temporally evolving information and evaluates models using temporal evidence, news accumulation, and structured external memory. Its knowledge-organization workflow extracts, indexes, and retrieves facts for answering questions.
2.1 Motivation
- Temporal question answering over evolving real-world information requires aggregating evidence across periods, where documents may give different answers.The paper uses a changing-team question as an example of this challenge.Source: §2.1 · p. 2
2.2 Datasets
- The benchmarks are Temporal Wiki, built on Templama, and Unified Clark, an extension of ERASE for real-world temporal retrieval using news.Temporal Wiki is publicly available, while Unified Clark is accessed through ERASE.Source: §2.2 · pp. 2–3
2.2.1 Temporal Wiki
- Temporal Wiki pairs Templama-derived questions with historical subject-page snapshots, using the earliest following-year snapshot or the closest later version when needed.Preprocessing removes structured elements and text artifacts; answer support is checked with surface matching and GPT-4o semantic verification.Source: §2.2.1 · p. 3
- Temporal Wiki tests closest, cumulative, and latest evidence: aligned snapshots may retain prior answers, cumulative snapshots aggregate through the question year, and latest snapshots use the most recent page.Cumulative evidence offers broader information coverage, but its text grows longer.Source: §2.2.1 · pp. 3–4, Figure 1 · p. 4
2.2.2 Unified Clark
- Unified Clark concatenates timestamped news articles for each entity and asks questions spanning multiple periods to test co-occurring, sometimes conflicting answers.Unlike cumulative Wikipedia pages, individual news articles generally reflect the answer current at publication.Source: §2.2.2 · p. 4
2.3 Experiment details
- Experiments use Llama 3.1 70B, Llama 3 8B, Mistral 7B Instruct v2, and Qwen-2 72B for evaluation, with Meta Prompting and LLM-evaluator consensus voting.RAG retrieves the top 12 chunks of 500 characters with 50-character overlap; Llama 3 8B is omitted from ICL because its context window cannot fit the full references.Source: §2.3 · pp. 4–5
2.4 Our knowledge organization approach
- The knowledge-organization workflow converts questions into subject-relation-object-timestamp quadruples, extracts related document facts, indexes them by subject and relation, then answers from matching entries.The framework builds a dynamic knowledge base so reasoning uses relevant structured knowledge rather than large, unfiltered context windows.Source: §2.4 · pp. 5–6, Figure 2 · p. 5
3 Experiments
The experiments compare zero-shot, ICL, RAG, and knowledge organization (KO) on Temporal Wiki and Unified Clark. Across the reported comparisons, KO performs best, while results also vary with context selection and task characteristics.
3.1.1 General observations
- RAG consistently outperforms ICL, and both benefit from shorter context; across models and question types, the closest snapshot beats the latest snapshot.Larger models generally achieve higher accuracy.Source: §3.1.1 · pp. 6–7
- For Temporal Wiki questions answered incorrectly zero-shot, KO uses external input best, followed by RAG; for initially correct questions, KO causes the least accuracy degradation.These categories distinguish questions improved by added supporting context from those whose accuracy falls when potentially conflicting or distracting context is introduced.Source: Table 1 · p. 6, §3.1 · p. 6, Table 2 · p. 7
3.1.2 Setup-specific observations
- Zero-shot accuracy tends to rise with fact-change frequency, while ICL accuracy fluctuates across document lengths without a consistent relationship to size.The authors hypothesize frequently updated pages may be more likely to have appeared in LLM training data.Source: §3.1.2 · p. 7
3.2 Unified Clark experiments
- On Unified Clark, KO outperforms all baselines, followed by RAG and ICL; the performance gap is more pronounced for complete-set time-sensitive answers.The task requires complete sets of time-sensitive answers rather than point predictions; Llama 3 8B ICL results are omitted because its context window cannot fit the full reference documents.Source: §3.2 · pp. 7–8, Table 3 · p. 8
4 Discussion
The paper evaluates question answering over temporally evolving knowledge and introduces Temporal Wiki and Unified Clark to assess robustness as facts accumulate, shift, or conflict. It reports that ICL and RAG struggle with outdated or conflicting information, while its knowledge organization framework structures and indexes facts for more reliable reasoning.
4 Discussion
- The knowledge organization framework incrementally structures and indexes facts, enabling more reliable reasoning over dynamic information.The paper concludes that structured temporal memory is important for future LLM systems.Source: §4 · p. 8
- RAG consistently outperforms ICL by filtering irrelevant context, but both approaches can struggle with temporal noise and underperform zero-shot baselines.Source: §4 · p. 8
- Temporal Wiki and Unified Clark assess robustness when facts accumulate, shift, or conflict over time.Source: §4 · p. 8
A Appendix
The appendix placeholder invites additional sections.
A Appendix
- The appendix placeholder says that additional sections may be included.Source: A Appendix · p. 12
Temporal Wiki Data Format
The dataset uses JSON records for events, with year-keyed incident objects that hold historical information and related question data.
Temporal Wiki Data Format
- Each JSON record represents an event and contains an event identifier plus year-keyed incident objects.The event identifier is unique, and incidents nest information under four-digit year keys.Source: Temporal Wiki Data Format · p. 12
- Each year-specific object records the question year, the information year used to find the answer, the question, and a list of correct answers.Source: Temporal Wiki Data Format · p. 12
- A year’s dump stores raw source data, including the Wikipedia URL, article body, and infobox; answer-comparison and model-response fields are reserved and null.Source: Temporal Wiki Data Format · p. 12