Source-linked AI summary
Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating
Heejin Choi
TL;DR
Continual-updating methods are often compared at one final checkpoint and one selected configuration, although the better operating point may depend on evaluation time and replay capacity. The paper compares a fixed periodic hierarchy with cumulative replay across time, rank, and query surface. The apparent winner changes across that evaluation region, so the authors propose reporting trajectories and capacity sweeps rather than declaring a robust winner when rankings vary.
Problem
Continual-updating methods are often compared using a final score under one selected configuration, treating method quality as a scalar.
Source: §1 · p. 1Method
The paper evaluates a fixed hierarchy against replay across evaluation time, replay rank, and query surface, then proposes reporting trajectories and capacity sweeps.
Source: §1 · p. 2, §10 · p. 10Results
At rank 72, time-averaged replay leads by approximately 9 points on training templates and 13 points on unseen paraphrases, although the final checkpoint suggests a tie.
Source: §1 · p. 2Takeaways & Limitations
The hierarchy is a lower-update-cost operating point, not a quality winner, and changing rankings call for reporting condition-dependent winner regions.
Source: §1 · p. 2The empirical reversal is limited to the evaluated time, rank, model, and query ranges; rank 72 is nominal rather than formally capacity- or compute-equivalent, and the stream is one domain dominated by P54.
Source: §10 · pp. 10–11
Abstract
from arXiv · showhide
Continual knowledge-updating methods are often declared superior from one final checkpoint and one conventional adapter rank. We show that this can be insufficient to identify the better operating point. Holding a periodic hierarchy fixed, we compare it with cumulative replay over a 24-month Wikidata stream while varying evaluation month, replay LoRA rank, and query formulation. The apparent winner changes across this region: on Qwen2.5-1.5B, the hierarchy's 5.0-point advantage over rank-8 replay becomes an 11.6-point deficit against rank-72 replay, and at high ranks a consolidation-aligned endpoint can suggest a tie while time-averaged replay leads by 9-13 points. The same rank-conditioned reversal appears on Llama-3.2-1B and held-out paraphrases. These results show that method ranking in continual updating can depend jointly on when performance is measured and how much replay-side adaptation capacity the baseline receives. We therefore propose reporting trajectories and capacity sweeps, and declaring a robust winner only when the ordering is stable across the evaluation region; otherwise, comparisons should report winner regions and retention-stability-cost frontiers. Under this protocol, the periodic hierarchy is a lower-update-cost operating point, not a quality winner.
1 Introduction
A single checkpoint and replay rank do not fully characterize continual-updating quality: the winner changes with evaluation time and replay capacity. The paper studies that joint reversal across model families and query surfaces, and frames the hierarchy as a lower-update-cost operating point rather than a quality winner.
1 Introduction
- Prior work studies boundary-only evaluation, editor rankings, rank-dependent factual storage, and stability–plasticity trade-offs; the paper’s narrower gap is their joint hierarchy-versus-replay reversal.The authors report finding no such study across both model family and query surface in their prior-art search through August 2026.Source: §1 · pp. 1–2
- At t=23, rank-8 cumulative replay leads the hierarchy by 4.9 points, but at t=24 the hierarchy leads by 13.1 points; at rank 72, time-averaged replay leads by approximately 9 points on templates and 13 on paraphrases.The t=23 and t=24 endpoints give opposite answers, while the rank-72 endpoint suggests a tie.Source: §1 · p. 2
- The hierarchy’s remaining contribution is lower update cost from replaying the full history only every K months, not evidence of better knowledge storage.The full rank sweep finds replay retention rises smoothly and saturates near rank 64; sufficiently capacitated replay exceeds the hierarchy in retention and unchanged-fact stability.Source: §1 · p. 2
- The principal analysis fixes the hierarchy at r64 slow plus r8 fast with K=6 and varies evaluation month, replay rank, and query surface.Replay rank shifts the average comparison level, while periodic consolidation adds temporal oscillation.Source: §1 · p. 2
- The rank-conditioned reversal replicates on Llama-3.2-1B and Qwen2.5-1.5B across training-template and held-out-paraphrase surfaces, motivating winner regions when rankings change.The proposed protocol predeclares evaluation times, replay capacities, and query surfaces.Source: §1 · p. 2
- The paper also reports scoped diagnostics: an O-LoRA residual trajectory on a relation-concentrated factual stream and a WISE contrast where P54 concentration raises answer concentration and lowers cumulative accessibility.The authors limit the WISE absolute result to their integration and note prior art for orthogonal-capacity exhaustion and immediate-versus-cumulative degradation.Source: §1 · p. 2
2 Benchmark and evaluation protocol
The benchmark follows 24 monthly Wikidata snapshots and evaluates retention and unchanged-fact stability over time. It compares sequential, replay, retrieval, hierarchy, and editing methods under specified query surfaces and seed settings.
2 Benchmark and evaluation protocol
- The stream contains 24 monthly Wikidata snapshots from 2024-01 to 2025-12, with up to 200 changed facts per month and 4,739 total selected facts.The unchanged control is fame-filtered and end-time-verified, and base-model cutoffs precede the stream.Source: §2 · pp. 2–3
- Scoring uses length-normalised loglikelihood multiple-choice against same-relation distractors, with retention and unchanged-control stability time-averaged over months 1–24 unless marked as endpoints.Each fact is scored on the training-template mem surface and unseen-paraphrase gen surface.Source: §2 · p. 3
- The methods include frozen and sequential LoRA baselines, full and partial replay, O-LoRA [Wang et al., 2023], RAG, two hierarchy variants, and WISE [Wang et al., 2024a].Three seeds are used for B2r8, B2r72, M1b, M2, and all Qwen/3B runs; intermediate ranks 16, 32, and 64 use one seed and are marked.Source: §2 · p. 3
- Every result JSON records model, method, ranks, seed, K, per-month costs, and completeness; headline tables are regenerated by script.The device comparison for remote versus local runs differs by at most 0.001 retention.Source: §2 · p. 3
3 Rankings form a time–replay-rank surface
The hierarchy’s consolidation-aligned endpoint can reverse or obscure the replay comparison. Across replay ranks, capacity shifts the average ordering while boundary alignment contributes a persistent positive endpoint bias.
3 Rankings form a time–replay-rank surface
- Boundary alignment drives the distortion: consolidation raises retention, so an endpoint at a consolidation peak overstates the comparison for the tested variants.M1b peaks at t ∈ {6, 12, 18, 24}; T=24 is a consolidation boundary.Source: §3 · pp. 3–4
- The endpoint margin is +0.131 versus a time-averaged margin of +0.040, an effect inflation of 3.32×.Source: §3 · p. 3
- For K=3/6/12, the endpoint is approximately 0.969 for all three variants, while stream means are 0.943/0.885/0.813.All three K values divide the 24-month stream, isolating timing across variants at identical capacity.Source: §3 · p. 3
- Across replay ranks 8/16/32/64/72, endpoint bias is +0.092/+0.094/+0.083/+0.085/+0.089 on templates and +0.113 to +0.081 on paraphrases, while hierarchy-winning months fall from 15/24 to 1.Capacity moves the average comparison level; consolidation adds a similarly sized positive bias across tested ranks.Source: §3 · pp. 3–4
- Replay leads by 4.9pp at T=23, while the hierarchy leads by 13.1pp at T=24; rankings flip in 9 of 24 months against rank-8 replay.Figure 1 marks replay-leading months and consolidation boundaries; its two candidate endpoints give opposite answers.Source: §3 · p. 3, Figure 1 · p. 4
- At ranks 16 and 32, the endpoint reverses the conclusion’s sign; at ranks 64 and 72, it reports a tie despite a nine-point replay win.At rank 72, the hierarchy leads in exactly one month of 24, t=6.Source: §3 · p. 4
4 Rank-conditioned reversal of the method comparison
The rank-8 comparison is not decisive: replay’s ordering against the fixed hierarchy changes with replay rank, while boundary-aligned evaluation adds a similar positive bias across tested ranks. The sweep also changes trainable parameter count and supports a lower-update-cost—not quality—advantage for the hierarchy.
4 Rank-conditioned reversal of the method comparison
- Replay rank shifts the average comparison level, while boundary-aligned timing adds a similar positive endpoint bias across tested ranks.Figure 2 plots endpoint and time-averaged margins against replay rank and their difference as endpoint bias.Source: Figure 2 · p. 5
- The sweep varies both update rank and trainable parameter count, so it is rank-conditioned rather than fully capacity- or compute-matched.Adapter scaling is held constant at α/r = 2.Source: §4 · p. 5
- Replay beats M1b on retention and stability from r=16 on training templates and from r=32 on held-out paraphrases, reversing the rank-8 conclusion.At r=16, gen retention is 0.797, still below M1b’s 0.814; the capacity sweep is time-averaged over months 1–24.Source: §4 · p. 5, Table 2 · p. 6
- Replay retention rises and saturates near r ≈64, while stability is unchanged from rank 8 to 16 and then falls as rank increases.Retention changes from mem 0.846 to 0.979 and gen 0.741 to 0.947; stability falls from mem 0.723 to 0.682 and gen 0.691 to 0.631 across the reported higher ranks.Source: §4 · p. 5
- The hierarchy’s supported advantage is approximately 3.5× lower update cost, not higher quality.The comparison does not claim rank is linearly additive in functional capacity.Source: §4 · p. 6
5 Resource accounting (supporting analysis)
The supporting accounting compares periodic cumulative replay with monthly full replay: periodic replay remains quadratic, although its work ratio approaches the consolidation period.
5 Resource accounting (supporting analysis)
- With period K, cumulative raw replay costs approximately T^2/2K + O(T), versus T^2/2 for monthly full replay.The ratio approaches K for fixed K as T grows.Source: §5 · p. 6
- At T=24, periodic replay costs 3.5× the reported work ratio, with a 6× asymptote for K=6.At T=12, the measured ratio is 2.5× versus 2.6× predicted; at T=24 it is 3.5× versus 3.6× predicted.Source: §5 · p. 6
- Bounded active adapter parameters do not bound replay-data storage; parameter-only merging targets that bounded-corpus property [Qiao and Mahdavi, 2025].The schedule remains quadratic with raw replay.Source: §5 · p. 6
6 Robustness: cross-model and cross-scale
The rank-conditioned reversal transfers across model families, while the reported scale evidence is limited to within-family comparisons. The section also includes stream-structured diagnostics and identifies untested model-rank settings.
6 Robustness: cross-model and cross-scale
- Within the Llama family, the margin shrinks 6× from 1B to 3B as hierarchy stability rises from 0.646 to 0.748.The Qwen result is a family-transfer check, not a third point on a scale trend.Source: §6 · pp. 6–7
- The rank-sweep figure shows replay retention crossing the fixed hierarchy, while stability stays flat through rank 16 and then falls.The hierarchy’s 3-seed band is shown for comparison.Source: Figure 3 · p. 7
- On Qwen2.5-1.5B, increasing replay rank from 8 to 72 flips the retention margin from +0.050 to −0.116 on mem and −0.109 on gen.At rank 72, replay also wins stability by +0.034; the hierarchy’s remaining advantage is 3.24× lower update cost.Source: §6 · p. 7, Table 3 · p. 7
- The rank-72 reversal was not tested at 3B, and intermediate replay ranks were not swept on Qwen.The Qwen test establishes family transfer of the reversal, not its dose-response shape.Source: §6 · p. 7
- O-LoRA’s residual grows approximately linearly through t=23, with R2=0.94; the final month is excluded because it has fewer training records.Figure 4 plots orthogonality residual against prior-task count.Source: Figure 4 · p. 8
7 Stress tests beyond immediate edit success
The stress tests measure cumulative behavior beyond immediate editing success, including O-LoRA residual overlap on a relation-concentrated factual stream and a controlled WISE contrast. Their results are scoped diagnostics, not universal mechanism claims.
O-LoRA
- O-LoRA’s time-averaged retention is 0.644 on mem and 0.594 on gen, below plain sequential LoRA’s mem retention of 0.657.From t=2 through t=23, residual overlap rises approximately linearly from 0.0020 to 0.0476, with slope 0.0020 per prior task and R2=0.94; month 24 is reported separately because it has fewer training records.Source: §7 · p. 8
- The O-LoRA trajectory is consistent with progressive subspace pressure in this setting, but does not establish a universal mechanism or rule out hyperparameter sensitivity.Month 24’s residual overlap is 0.2017, but it contains 147 training records rather than approximately 194–200 and is reported separately.Source: §7 · p. 8
WISE
- Immediate rewrite accuracy is 1.000 after each edit, while cumulative multiple-choice retention remains near the frozen base; P54-only editing shows greater answer collapse than the six-relation arm.The P54-only arm has top-attractor share 0.800, 12 distinct answers, and cumulative MC 0.330, versus 0.585, 45, and 0.390 for the six-relation arm.Source: §7 · p. 8
Scope
- These stress tests support cumulative, stream-structured evaluation; relation-concentrated operational streams differ from more diverse standard edit benchmarks.Prior sequence-editing work reports effects of target diversity and sequence length [Huang et al., 2024].Source: §7 · p. 8
8 Secondary result: the index-size/retention trade-off in hybrids
The hybrid comparisons expose distinct quality, index-size, stability, and latency trade-offs rather than a single dominant store. Restricting retrieval to the current consolidation window sharply reduces index size but lowers retention.
Store comparison
Hybrid variants
- 0.988 ± .002 retention with an 11,551-entry bounded index belongs to old→RAG, recent→adapter; old→adapter, recent→RAG yields 0.746 ± .000 and 147.Source: §8 · p. 9
Bounded retrieval
- Restricting retrieval to the current consolidation window shrinks the index 80× but costs 24 points of retention because the consolidated adapter must serve the entire past.Membership routing raises stability from 0.475 to 0.623, reaches 99% of oracle end-task performance, and merged adapters serve at base latency (19.5 vs 21.3 ms/query).Source: §8 · p. 9
9 Related work and novelty boundary
Prior work already establishes evaluation-dependent rankings, rank effects, cumulative-accessibility failures, and orthogonal-subspace exhaustion. The paper’s narrower novelty is the joint update-time by replay-rank analysis and its resulting winner reversal.
Evaluation-dependent rankings
- Boundary-only evaluation can hide temporary forgetting and recovery [De Lange et al., 2023], while knowledge-editor rankings vary with metrics, methodology, and edit batch size [Pohl et al., 2025].The paper frames its narrower contribution as joint update-time by replay-rank analysis with concrete endpoint error regimes and cross-family replication.Source: §9 · p. 9
Rank and parametric memory
- Rank-dependent factual-memory saturation and rank–forgetting frontiers are established; the distinct result is that rank-8 and higher-rank replay reverse the hierarchy comparison across two model families and query surfaces.Related work includes Back et al. [2026], Jiang et al. [2024], and Lu et al. [2026].Source: §9 · p. 9
Lifelong editing
- Sequential editing research already studies retention, generalization, locality, and memory organization, including immediate-versus-cumulative performance and temporal-state collapse.The paper characterizes its WISE analysis as a scoped quantitative stress test, not a new general failure principle [Huang et al., 2024; O’Neill, 2026; Liu et al., 2026].Source: §9 · p. 9
Orthogonal subspaces
- Orthogonal-subspace exhaustion is not the novelty: O-LoRA limits interference with orthogonal updates [Wang et al., 2023], and E2-LoRA discusses capacity exhaustion [Li et al., 2026].The paper’s contribution is a residual-curve measurement in this stream.Source: §9 · p. 10
Temporal data and consolidation
- Temporal factual streams and fast/slow memory architectures also have precedents; the claimed benchmark contribution is the joint update-time by replay-rank protocol for a fixed hierarchy and its winner reversal.The cited work includes TemporalWiki [Jang et al., 2022], ForgetBench [Gu et al., 2026], and OAKS [Kim et al., 2026].Source: §9 · p. 10
10 From winner reversal to a robust evaluation protocol
With the hierarchy fixed, update phase and replay rank can change which method appears to win. The paper proposes claiming superiority only when the ordering stays stable across a predeclared evaluation region; otherwise, report winner regions and a Pareto frontier.
10 From winner reversal to a robust evaluation protocol
- A robust superiority claim requires the signed margin to keep the same sign across predeclared evaluation times, baseline capacities, and query formulations, with seed uncertainty reported at headline comparisons.When the sign changes, report a winner map or retention–stability–cost frontier instead of a universal scalar winner claim.Source: §10 · p. 10
- Evaluate throughout the stream, disclose event-aligned endpoints, sweep baseline capacity, test training and held-out formulations, and report retention, stability, update work, storage, and serving cost separately.The authors do not collapse these dimensions into a single metric because deployment preferences differ.Source: §10 · p. 10
- The reversal is bounded to evaluated time, rank, model, and query ranges; intermediate Llama ranks are single-seed, while intermediate Qwen ranks and rank-72 replay at 3B remain untested.Rank 72 is a nominal comparison point, not a formal capacity- or compute-equivalence to the hierarchy.Source: §10 · p. 10
- The stream covers one domain dominated by P54 and lacks full versioned semantics for repeated changes; gen tests unseen paraphrases, not unrestricted generation or multi-hop use.The WISE relation contrast also does not fully match answer entropy, relation difficulty, or target perplexity, and its absolute performance is scoped to the EasyEdit integration.Source: §10 · pp. 10–11
- The O-LoRA final-month spike is confounded with a smaller training batch, limiting attribution to the method.Source: §10 · p. 11
- The hierarchy-versus-replay winner changes with update phase and replay rank, so a single endpoint at one default rank can inflate, reverse, or hide the comparison.The reversal also appears on held-out paraphrases and a second model family.Source: §10 · p. 11