Source-linked AI summary
Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
Jiyeon Kim, Hyunji Lee, Dylan Zhou, Sue Hyun Park, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Sungmin Cha, Minjoon Seo
TL;DR
LLMs need to adapt as real-world knowledge changes continuously, but existing benchmarks mainly test static or offline settings. The paper introduces OAKS, a streaming benchmark with fine-grained evolving context and interval annotations, and finds that 14 models and memory strategies remain unreliable, especially under frequent updates. The study also identifies state-tracking delays, distraction, and degradation at later intervals, while noting limits on model coverage and context assumptions.
Problem
Current benchmarks primarily target static knowledge or offline tasks, leaving online adaptation to dynamically changing information inadequately evaluated.
Method
OAKS evaluates online adaptation using OAKS-BABI and OAKS-Novel, where incrementally streamed context chunks contain evolving facts and answers are annotated at each interval.
Results
Across 14 models, average accuracy is 39.4% on OAKS-B and 57.5% on OAKS-N, with models struggling especially under frequent knowledge updates.
Takeaways & Limitations
State-of-the-art models and agentic memory systems remain challenged by streaming updates, showing delays in state tracking, distraction, and later-interval degradation.
Takeaways & Limitations
The study evaluates 14 representative models and focuses on inference-time adaptation through incremental context accumulation.
Abstract
from arXiv · showhide
LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.
1 Introduction
OAKS addresses the gap between dynamic, incrementally arriving knowledge and benchmarks that mainly test static or offline settings. It evaluates online state tracking through evolving context streams and finds substantial adaptation failures across models and inference strategies.
- OAKS evaluates online adaptation to streaming, continually updating knowledge that may supersede or contradict prior facts.
- The benchmark uses OAKS-BABI and OAKS-Novel, with context chunks and answers annotated at each interval to capture answer transitions.
- 14 models average 39.4% accuracy on OAKS-B and 57.5% on OAKS-N, with lower accuracies of 33.3% and 53.0% under frequent updates.
- Thinking mode improves adaptability and stability, while models show over-updating, under-updating, distraction, missed phases, and degradation at later or reasoning-intensive intervals.
2 Related Work
Prior work studies continual knowledge, state tracking, and online adaptation, but OAKS targets fine-grained updates to open-ended knowledge states over long horizons.
- Existing continual-knowledge benchmarks often use few updates or divergent facts rather than repeated updates to the same underlying fact.
- OAKS differs from short-term state-tracking benchmarks by evaluating open-ended knowledge states and temporal consistency across continuous updates.
- Prior online or lifelong-learning work emphasizes acquiring capabilities or tracking synthetic facts with limited update complexity, whereas OAKS focuses on fine-grained knowledge updates.
3 ONLINE ADAPTATION TO CONTINUAL KNOWLEDGE STREAMS
OAKS turns continually changing knowledge into a stepwise online evaluation: models repeatedly answer fixed questions as context chunks accumulate. Its datasets annotate evolving answers and support both retrieval and multi-chunk reasoning demands.
- Dataset: OAKS-BABI and OAKS-Novel provide sequential chunks containing knowledge updates and questions targeting facts that evolve over time.
- OAKS-BABI: OAKS-B reformulates BABILong with state-change questions and interval annotations, containing 1.2k questions, 128k-token contexts, 65 chunks, and 4.7 answer changes per question.
- OAKS-Novel: OAKS-N uses 39 full-length novels with dynamically interacting narratives and human-curated questions.
- Setup: At each interval, the model answers the same questions using all observed chunks, enabling evaluation of updating, incorporating, and retaining knowledge online.
- Metric: Interval-level accuracy compares each prediction with the current ground-truth answer, then averages scores across intervals and the dataset.
4 Experimental Setup
The experiments evaluate 14 open-source and proprietary LLMs using concatenated context, retrieval, and agentic-memory representations. Inference is conducted with standardized decoding settings across the tested configurations.
- Base Models: The study evaluates 14 LLMs spanning open-source and proprietary families and a wide range of model scales.
- Context Representations: The Base setting concatenates preceding chunks, RAG retrieves up to 30 relevant earlier chunks, and agentic memory systems provide an additional context representation.
- Inference Setup: Experiments mostly use 8 A100 80G GPUs with temperature 0.7, TopP 0.8, TopK 20, and MinP 0.
5 Evaluation Result
OAKS remains challenging across models and inference strategies, with performance worsening as answer transitions become more frequent. Larger or stronger base models and thinking mode generally help, while naive RAG offers little benefit and agentic memory systems remain limited.
- Overall performance: 33.0% average accuracy on OAKS-B and 52.9% on OAKS-N for open-source models, compared with 60.9% and 72.6% for closed-source models.Even Gemini 3 Pro reaches only 66.3% on OAKS-B and 75.5% on OAKS-N.
- Overall performance: Performance generally scales with model size within the same model family, while stronger base models and proprietary models tend to achieve higher accuracy.Among similarly sized models, Qwen3 generally outperforms alternatives; MoE models are comparable to dense models of similar scale.
- Transition frequency: 42.2% on Sparse, 40.6% on Moderate, and 33.3% on Frequent OAKS-B questions; OAKS-N averages 65.4%, 57.2%, and 53.0%, respectively.Frequent updates require repeated state revisions while retaining previously valid information, increasing tracking and retention difficulty.
- Context strategies: Naive RAG changes average performance by +0.4% on Sparse, -0.04% on Moderate, and -0.8% on Frequent subsets.The authors identify retrieval difficulty and sensitivity to complex or irrelevant context as challenges in streaming settings.
- Inference strategies: Thinking mode consistently improves OAKS performance, with the largest reported gain occurring on bridge questions at 15.4%.Bridge questions require reasoning over multiple pieces of evidence distributed across non-contiguous chunks.
- Agentic memory: Agentic memory methods underperform naive RAG overall, although MemAgent achieves the best results on the Moderate and Frequent subsets.Its interval-based tracking objective aligns with the benchmark, but training rewards are computed only after all chunks rather than at each interval.
6 Analysis
The analysis shows that models vary in how they update predictions, with thinking mode improving transition timing and correctness. Errors also arise from unnecessary updates, missed phases, distraction, and degradation over later intervals.
- Predicted transition behavior: Thinking mode increases Adaptability and Stability across Qwen3 and Gemini 2.5, while non-thinking models more often retain outdated answers.Non-thinking variants show more Prescience, Stubbornness, and Obstinacy.
- Predicted transition behavior: GPT-OSS and Qwen3 change predictions more often, whereas Gemini 2.5 and Gemma 3 stay more often, producing Volatility and Obstinacy as dominant error modes.The average change rates are 63.2% versus 44.2%, while stay rates are 36.8% versus 55.8%, respectively.
- Predicted transition behavior: 38.1% of intervals under GT Phase Transitions show Adaptability, but Volatility dominates No GT Transition intervals at 32.1%.The pattern indicates frequent detection of true transitions alongside unnecessary updates when facts remain unchanged.
- Intra-phase behavior: Intra-phase evaluation separates errors into Acquisition Latency, Distraction Susceptibility, and Phase Miss rate.These metrics respectively measure delayed correct prediction, incorrect predictions after an initial correct prediction, and entirely missed phases.
- Intra-phase behavior: Gemma 3 and GPT-OSS can have comparable accuracy while differing in error structure: Gemma 3 has higher Phase Miss, whereas GPT-OSS has higher Acquisition Latency and Distraction Susceptibility.GPT-OSS captures each phase at least once despite greater volatility.
- Question types and temporal degradation: Bridge questions have the highest Distraction Susceptibility, and accuracy declines at later intervals, especially for bridge and tracking questions in OAKS-B.The later-interval decline is linked to missed evidence persisting and accumulating across subsequent intervals.
7 Conclusion
The paper introduces OAKS, a benchmark and two datasets for testing online adaptation to continually changing knowledge. Experiments show that even advanced models and agentic memory systems struggle under frequent updates and long streaming contexts.
- Contribution: OAKS evaluates online adaptation to streaming, continually evolving knowledge using incrementally revealed context and interval-specific answer annotations.The benchmark asks whether models maintain the correct state as facts change over time.
- Datasets: OAKS-BABI and OAKS-Novel contain long sequences of small context chunks with fixed questions and answers annotated at every interval.This design tracks evolving facts throughout the stream.
- Findings: Experiments on 14 models show persistent difficulty, particularly with frequent updates and later intervals, as models are distracted and lose previously identified states.The conclusion includes both state-tracking failures and distraction from surrounding context.
Limitations
The study is constrained by evaluation cost, model coverage, dataset complexity, and reliance on inference-time context accumulation. The authors identify broader architectures, harder natural texts, prior-free contexts, and parametric online learning as future directions.
- Scope and cost: Extensive inference requirements limited the study to 14 representative models because of computational and API costs.The authors propose testing a broader range of architectures to improve generalization of the findings.
- Dataset scope: The current datasets do not fully cover more complex natural texts with frequent or complex fact transitions.The authors identify these settings as needed to study scalability and error accumulation further.
- Adaptation setting: Contexts may contain the models’ parametric knowledge priors, and the analysis focuses on inference-time adaptation rather than updating internal weights.Prior-free contexts and parametric online learning are proposed as complementary future directions.
B.1.1 Data statistics
OAKS combines synthetic and literary datasets designed to measure answer changes across sequential context updates. Questions are curated and annotated at each state transition, with evaluation organized by answer-change frequency.
- Answer-change frequency: Questions are partitioned into approximately balanced subsets by answer-change frequency, while keeping questions with identical change counts together.OAKS-B uses bins of 2–3, 4–5, and 6–20 changes; OAKS-N uses 2–3, 4, and 5–19 changes.
- OAKS-BABI: OAKS-B uses fact-based narratives with questions targeting multiple knowledge transitions and four question types: tracking, comparison, bridge, and counting.The dataset averages 87 facts per example and 4.7 answer changes per question.
- OAKS-BABI: OAKS-B contains 1.2k questions across 65 chunks of 2k tokens, with an average of 87 facts per context.
- OAKS-Novel: OAKS-Novel uses literary narratives with evolving entities, flashbacks, temporal jumps, and concurrent plotlines to create naturalistic state-tracking questions.
- OAKS-Novel: OAKS-N questions were manually filtered, revised, and evidence-annotated so exactly one answer option is correct at each narrative chunk.Only 55% of initial questions were retained or reformulated as high-quality.
C Experimental Setup
The experiments evaluate 14 open and proprietary models on sequential-context tracking using multiple context-construction and retrieval strategies. The setup includes baseline, RAG, rolling-window, and combined approaches under long-context constraints.
- Models: The benchmark evaluates 14 language models spanning open-source and proprietary systems and a wide range of model scales.
- Retrieval: Retrieval uses only chunks from previous intervals and preserves their temporal order, with 30 relevant memory chunks retrieved by default.
- Context construction: Under long-context constraints, rolling windows retain recent chunks, while RAG+RW combines retrieved history with recent context.
- RAG: RAG indexes chunks with explicit indices and prepends the top retrieved fragments to the question prompt.
- Inference: Thinking mode improves OAKS-B performance but degrades performance on OAKS-N, whose naturalistic narratives require implicit multi-hop reasoning.
D.2 Analysis on Performance of RAG and RW
Context construction strategies generally improve performance over baseline, but no single strategy dominates across both datasets. RAG benefits OAKS-B, whereas OAKS-N benefits from combining retrieval with a rolling window.
- Overall comparison: No single context-construction strategy consistently dominates across OAKS-B and OAKS-N.
- OAKS-B: RAG achieves the best performance on OAKS-B, improving accuracy by +2.0%.
- OAKS-N: RAG decreases OAKS-N performance by -1.8%, while RAG+RW achieves the best result with a +1.8% improvement.
- Chunk count: Increasing the number of retrieved or windowed chunks initially improves performance, with scores peaking around 30 chunks.
D.3 Difference by dataset
Performance differs substantially between the synthetic OAKS-B and literary OAKS-N datasets. OAKS-N scores higher, while the authors attribute the gap to answer format, prior-knowledge effects, and task difficulty.
- Dataset comparison: 39.4% average accuracy on OAKS-B is lower than 57.5% on OAKS-N.
- Possible explanations: The authors conjecture that OAKS-N’s multiple-choice format contributes to its higher performance by restricting the answer space.
- Possible explanations: OAKS-N narratives may remain susceptible to prior-knowledge bias, while OAKS-B may be harder because its synthetic design enables intrinsically more complex tasks.
E.1 Fine-grained Analysis of Predicted Knowledge Transition Behavior
Models can often identify whether knowledge should change, but producing the correct updated answer is harder. Errors reflect distinct transition behaviors, including unnecessary changes, inertia, and broader mixtures of maladaptive responses.
- Transition prediction and answer correctness: 31.2% of cases showed correct transition behavior versus 18.8% incorrect, while answer correctness was 22.8% versus 27.2% incorrect.The results indicate that recognizing whether a transition occurs is easier than determining the updated answer.
- Transition prediction and answer correctness: Among correctly answered cases with correct transition predictions, adaptability occurred more often than stability.The analysis identifies maintaining stable knowledge as more challenging than updating at the correct time.
- Failure modes: Incorrect answers were associated with volatility when transitions were predicted incorrectly and maladaptation when transitions were predicted correctly.These behaviors correspond to unnecessary or poorly timed changes.
- Failure modes: The gap between changing and staying behaviors was 7.5 for incorrect answers versus 14.6 for correct answers.The smaller gap among incorrect cases suggests that errors arise from a broader range of failure modes.