PaperlayerView exact revision on arXiv

Source-linked AI summary

ChurnBench: A Drift-Aware Benchmark Demonstrating That Refresh Scheduling, Not Cache Age, Governs Staleness in Agentic AI

Vivek Kumar Singh, Preeti Priyam

arXiv:2609.11515v1cs.SE

TL;DR

Existing retrieval evaluations freeze enterprise data, leaving them unable to measure whether an agent’s answer remains true after the underlying world changes. ChurnBench generates a drifting multi-source enterprise fabric with ledger-derived gold answers and a timestamped freshness-error protocol, revealing that scheduled refresh and TTL—not cache age—govern observed staleness.

  • Problem

    Frozen retrieval and agent benchmarks cannot measure stale answers because their corpora do not change between retrieval and evaluation.

  • Method

    ChurnBench generates a timeline of enterprise changes, projects live stores from an append-only ledger, and compares ledger-derived gold at retrieval and evaluation timestamps.

  • Results

    Scheduled refresh makes staleness insensitive to cache age, while disabling tiered refresh raises freshness errors from 4 to 45 at 28 days and leaves them unchanged over one day.

  • Takeaways & Limitations

    Drift benchmarks should vary TTL configuration against entity change rates and record per-entity refresh timestamps rather than sweep drift-window length.

  • Takeaways & Limitations

    The evaluation uses one model, one agent framework, one domain, synthetic data, and no baseline or cross-provider comparison.

Abstract

from arXiv · show

In production, agentic systems answer questions over data that lives in several places and keeps changing: licenses are reassigned, users offboarded, prices changed, contracts renewed. Existing retrieval benchmarks freeze the data, so they cannot ask whether an agent's answer is still true, only whether it found the right passage. We present ChurnBench, an open-source benchmark that generates a four-source enterprise data fabric as a timeline rather than a snapshot. Every change is written to an append-only ground-truth ledger, and gold answers are computed from that ledger, never from the live stores. An answer that was correct when its data was retrieved but wrong when evaluated is therefore detected and labeled a freshness error, distinct from a reasoning error; we validate this by resolving ground truth at both timestamps for every case reported. Using the instrument, we find that when a system refreshes on a schedule, cache age does not predict staleness. Across cache ages of 1, 14, and 28 days, freshness errors were 7, 4, and 4, because scheduled refresh bounds staleness by time-to-live, and no TTL lapse was observed in any window. A controlled ablation confirms the mechanism: disabling tiered refresh raises freshness errors from 4 to 45 at 28 days and leaves them identical at one day. The variable a drift benchmark should sweep is therefore TTL configuration against each entity's rate of change, not drift-window length. ChurnBench, the evaluation harness, and all per-error data are released open source.

ORCID 0009-0002-9350-3207

ChurnBench addresses the inability of frozen evaluations to measure whether agent answers remain true as enterprise data changes. It introduces a timeline-based benchmark and uses it to show that scheduled refresh, rather than cache age, governs staleness.

  • Frozen retrieval benchmarks cannot measure stale answers because their corpora do not change between retrieval and evaluation.
  • ChurnBench generates enterprise activity as a timeline, folds an append-only ledger into live stores, and resolves gold answers from the ledger.
  • Freshness errors are separated from reasoning errors by comparing gold answers at evaluation time and effective retrieval time.
  • Scheduled refresh bounds staleness by time-to-live, so cache age does not determine the age of retrieved entities.
  • The benchmark recommends sweeping TTL configuration against entity change rates and recording per-entity refresh timestamps rather than drift-window length.
  • The paper contributes ChurnBench, a verified freshness-error protocol, and a controlled finding that tiered refresh scheduling governs observed staleness.

II. RELATED WORK

Prior agent and retrieval benchmarks generally evaluate systems over static, single-source or pinned corpora. ChurnBench instead combines heterogeneous enterprise sources with controlled temporal drift and measures freshness in the grounding layer.

  • Existing retrieval and enterprise agent benchmarks evaluate agents against fixed snapshots, without injecting drift or measuring freshness.
  • ChurnBench differs by combining several heterogeneous enterprise sources across modalities with a world that changes between indexing and evaluation.
  • Prior temporal-drift work studies changing textual facts or agent memory, whereas ChurnBench targets caches, aggregates, and indexes over enterprise systems.
  • ChurnBench shares MemStrata’s bi-temporal-ledger idea but applies it to grounding-layer freshness rather than prose facts in a single corpus.

III. THE SYSTEM UNDER MEASUREMENT

The measured system places a deterministic grounding layer between heterogeneous sources and the agent runtime. Its registry makes source locations, freshness policies, capabilities, and routing decisions explicit and auditable.

  • The grounding layer sits between heterogeneous sources and the agent runtime, providing the pipeline measured by ChurnBench.
  • A declarative registry records each entity class’s source, staged location, freshness tier, TTL, supported measures, and last-refresh time.
  • The registry makes routing deterministic and coverage auditable because each measure is either registered or absent.
  • A rule-based router selects among staged queries, federated joins, live warehouse queries, origin-interface calls, and vector lookup.
  • The router’s recorded path supports the freshness-error protocol by identifying how and when each retrieved fact was served.

C. Tiered Refresh

ChurnBench models enterprise data as a changing, multi-source fabric and refreshes entity classes according to tier-specific schedules. This design enables controlled testing of whether refresh scheduling, rather than cache age, governs staleness.

  • Each entity class is refreshed when its tier-specific TTL lapses, allowing fast-changing entities to refresh more often than stable ones.
  • The benchmark uses four enterprise sources spanning historical warehouse facts, operational state, SaaS interfaces, and contract documents.
  • Software asset management provides naturally drifting events including reassignment, offboarding, renewal, and repricing across structured and unstructured data.
  • A deterministic simulator generates day-by-day mutations from event streams and writes them to an append-only ledger, with identical seeds producing identical ledgers.
  • Live stores are timestamped projections of the ledger, while non-model gold answers are resolved directly from the ledger to prevent projection or grader errors from changing correctness.

C. Frozen-at-T Evaluation

The benchmark fixes cache construction at T′ and evaluation at T, while deriving gold answers from an append-only ledger at both timestamps. It measures freshness errors separately from reasoning failures and preserves two task types that current-state stores cannot answer.

  • Each run builds caches and indexes at T′, asks questions at T, and treats the gap as cache age.Freezing the world during a run sacrifices mid-run drift for exact reproducibility.
  • The 180 generated tasks span single-source, cross-source, and cross-modality questions across four executive intents.The difficulty tiers contain 81, 63, and 36 tasks, respectively.
  • Event-history questions and per-license cost totals remain outside the benchmark’s current-state grounding architecture because the required historical or per-license data is absent.
  • Freshness error means the answer is wrong against gold at T but correct against gold at its effective retrieval time.The effective retrieval time is derived from the trace, using T for live routes and an entity’s last-refresh timestamp for staged routes.
  • Ground truth is resolved at both timestamps for every reported freshness error, excluding answers that merely match an earlier world state.

V. EXPERIMENTAL SETUP

The experiments use a fixed software and model configuration, a large event ledger, and reproducible run artifacts. Retries address interface instability, while the final design intentionally lacks a baseline comparison.

  • All runs use one model and agent framework, with the three-window sweep reported separately because it used reasoning mode enabled.The 28-day freshness-error count was the same under both reasoning-mode settings.
  • The ledger contains 56,370 events, including dense consumption activity and comparatively rare contract renewals.This imbalance matters for the later attribution analysis.
  • Interface-failed tasks were rerun under a hardened retry policy, with unchanged results preserved and provenance recorded.
  • No baseline comparison was run under the final design, so the reported accuracy describes one implementation under two refresh settings.Development-only arms were not evaluated on the 180-task set under the reported configuration.
  • The repository commits the ledger, generation seed, configuration metadata, per-error timestamps, and per-statistic files needed to regenerate results.Freshness-error checks can be rerun without making new model calls.

VI. RESULTS

The controlled comparison shows that refresh scheduling, rather than cache age alone, governs freshness errors. Scheduled refresh keeps entity ages bounded, whereas disabling it creates substantial long-window staleness.

  • The experiment holds model, task set, seed, fabric, prompts, reasoning mode, and other factors fixed while varying refresh setting and cache age.Every reported freshness error passed the paper’s timestamp-based definitional check.
  • 11-fold more freshness errors occur with refresh disabled at 28 days, while both configurations produce exactly 7 errors at one day.The short-window errors come from the same seven tasks and entity classes; the long-window contrast isolates refresh behavior.
  • With tiering enabled, one-day-TTL entities remain within one day of evaluation and seven-day-TTL entities remain one to six days old at a 28-day cache age.With tiering disabled, every entity’s last refresh equals cache build time, producing 29-day staleness.
  • Contract terms remain unchanged because their 30-day TTL does not lapse within the 28-day window.

B. Why the Cache-Age Sweep Fails

The cache-age sweep fails because scheduled refresh prevents cache-build age from reaching the retrieval layer for shorter-TTL entities. The benchmark should instead vary TTL configuration against entity change rates.

  • No TTL lapse occurs in any tested window because scheduled refresh renews each entity before its tier expires.Only 30-day contract terms retain the original build time within the widest window.
  • A drift benchmark should sweep TTL configuration against entity change rates while holding cache age fixed, rather than widening the build-to-evaluation window.
  • Figure 2 shows scheduled refresh resetting each class’s age within its TTL, while disabled refresh makes all tiers age linearly.The hot tier contributes 25 of 45 errors only when refresh is disabled.

C. Where Staleness Actually Lands

Freshness errors concentrate in entity classes where mutation rates and refresh tiers create exposure, while removing scheduled refresh exposes fast-changing entities. The results motivate measuring per-entity refresh timestamps and matching tiers to observed mutation rates.

  • Error attribution: 14 of 15 tiered-refresh freshness errors came from seven-day price entities, while one came from contract terms.The price errors comprised nine cost-center spend totals and five highest-spending cost-center queries.
  • Error attribution: Scheduled refresh produced zero errors for one-day-tier entities and cold contract entities because their exposure to coincident mutations was limited.Contract terms had 27 renewals across 56,370 events, so their long tier was stale but rarely overlapped a query and mutation.
  • Effect of removing refresh: 45 freshness errors without refresh at 28 days included 15 user-status errors, 10 assignment errors, and 20 price errors.User status and assignments produced zero errors under tiered refresh, whereas prices already contributed errors with tiering enabled.
  • Implication: Staleness exposure depends on tier width and entity mutation rate together, so tiers should be assigned from measured mutation rates.The widest tier produced no errors because its entities changed rarely, while fast-changing entities appeared after refresh was removed.
  • Measurement caveat: TTL-lapse counts were zero under both refresh settings for opposite reasons, making per-entity refresh timestamps the reliable instrument.Tiered refresh renews entities before expiry; disabled refresh skips expiry checks entirely.

D. Accuracy by Task Tier, and the Effect of Coverage

Accuracy varies by task tier and is strongly shaped by semantic-registry coverage rather than serving architecture alone. The paper therefore treats these accuracy results as implementation-dependent, with additional limits from its narrow experimental scope and small freshness-error counts.

  • Accuracy by task tier: Tier 3 accuracy was nearly unchanged between settings because both configurations rebuild the document index at T′.These tasks reach contract prose through the document index, which is rebuilt at the same time in both settings.
  • Accuracy by task tier: The task tiers discriminate as intended: single-source questions are answered more often than cross-source questions, and disabling refresh costs most on tiers touching stageable entities.This pattern is reported in the matched 28-day comparison.
  • Effect of coverage: 109 of 180 tasks target unregistered measures, and 49 to 53 of the 51 to 69 reasoning errors per run come from those tasks.Reported accuracy therefore largely measures registry coverage, not the architecture.
  • Statistical power: Tiered-refresh freshness-error counts range only from four to seven per 180-task run, although the 4-versus-45 ablation gap is much larger.The paper relies on direct mechanism measurements in addition to the small-sample sweep.
  • Scope: The benchmark covers one model, one agent framework, one domain, and synthetic data, with no baseline or cross-provider comparison.Synthetic generation enables controlled drift and exact ground truth, but prevalence claims do not transfer directly to real estates.

VIII. DISCUSSION AND FUTURE WORK

ChurnBench shows that tiered refresh makes staleness depend on TTL and entity change rates rather than cache age or drift-window length. The authors recommend rate-matched tiering and identify further experiments needed to quantify cost, test transfer, and compare grounding architectures.

  • VIII. Discussion and Future Work: Tier assignment should match each entity's measured rate of change: rapidly changing entities need shorter tiers, while slowly changing entities can use longer ones.The paper reports that only entities whose tier width overlapped their change rate produced staleness, while the widest tier was harmless for rarely changing entities.
  • VIII. Discussion and Future Work: Drift benchmarks should sweep TTL configuration against entity change rates rather than drift-window length, which produced a flat outcome under maintained indexes.The authors recommend recording per-entity refresh timestamps as well as sweeping TTL settings.
  • VIII. Discussion and Future Work: The next experiment is to hold cache age fixed, vary tier width across a grid, and measure the staleness-versus-cost tradeoff directly.This would turn the current qualitative guidance into a quantitative evaluation.
  • VIII. Discussion and Future Work: Open questions include whether tiering transfers beyond software asset management and how staged, hierarchical, and live-only grounding architectures trade staleness against cost.The paper identifies these as future directions rather than settled findings.
  • VIII. Discussion and Future Work: Ablation raises freshness errors from 4 to 45 when tiered refresh is disabled, showing the scheduler's central role in controlling staleness.The reported comparison isolates disabling the scheduler as the intervention.
Loading 2609.11515v1…